From victor.kryukov at gmail.com Fri Jan 4 07:12:00 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Fri, 4 Jan 2008 01:12:00 -0600 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. Message-ID: Hello list, I've decided to put elephant 0.9.1 under some heavy load test, and play with Netflix data set a little bit. The attached program that tries to import everything in BerkeleyDB fails when trying to import movie file number 8 with the following traceback. Do you have any idea what could be the problem? Thanks, Victor. Berkeley DB error: Cannot allocate memory [Condition of type BDB-DB-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [ABORT] Exit debugger, returning to top level. Backtrace: 0: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T DB-BDB::BDB-BTREE)) #) 1: ((LAMBDA NIL)) 2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER #1="#<...>" . #1#)) # # # #) 3: (IMPORT-ALL-MOVIES) 4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (IMPORT-ALL-MOVIES) #) 5: (SWANK::EVAL-REGION "(import-all-movies) " T) 6: ((LAMBDA NIL)) 7: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-SYNTAX-HOOKS (T)) # # #) 8: (SWANK::CALL-WITH-BUFFER-SYNTAX #) 9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(import-all-movies) ") #) 10: ((LAMBDA NIL)) 11: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 12: ((LAMBDA NIL)) 13: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T T)) # # # #) 14: (SWANK::CALL-WITH-REDIRECTED-IO # #) 15: (SWANK::CALL-WITH-CONNECTION # #) 16: (SWANK::HANDLE-REQUEST #) 17: (SWANK::PROCESS-AVAILABLE-INPUT # #) 18: ((FLET SWANK::HANDLER)) 19: ((LAMBDA (SWANK-BACKEND::_)) #) 20: (SB-IMPL::SUB-SERVE-EVENT NIL NIL NIL) 21: (SB-SYS:WAIT-UNTIL-FD-USABLE 0 :INPUT NIL) 22: (SB-IMPL::REFILL-INPUT-BUFFER #) 23: (SB-IMPL::INPUT-CHAR/UTF-8 # NIL #:EOF-OBJECT) 24: (READ-CHAR # NIL #:EOF-OBJECT #) 25: (READ-CHAR # NIL #:EOF-OBJECT #) 26: (READ-CHAR # NIL #:EOF-OBJECT #) 27: (READ-PRESERVING-WHITESPACE # NIL (NIL) T) 28: (READ-PRESERVING-WHITESPACE # NIL (NIL) NIL) 29: (READ # NIL (NIL) NIL) 30: (SB-IMPL::REPL-READ-FORM-FUN # #) 31: (SB-IMPL::REPL-FUN NIL) 32: (SB-IMPL::REPL-FUN NIL) 33: ((LAMBDA NIL)) 34: ((LAMBDA NIL)) 35: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) 36: (SB-IMPL::TOPLEVEL-REPL NIL) 37: (SB-IMPL::TOPLEVEL-INIT) 38: ((LABELS SB-IMPL::RESTART-LISP)) -- Yours Sincerely, Victor Kryukov Chicago GSB class of 2008 773-618-9501 From victor.kryukov at gmail.com Fri Jan 4 07:14:03 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Fri, 4 Jan 2008 01:14:03 -0600 Subject: [elephant-devel] Re: Berkeley DB error: Cannot allocate memory. In-Reply-To: References: Message-ID: This is the program: ;;;; ;;;; Playing with Netflix dataset using Elephant to store data ;;;; (mapc (lambda (x) (asdf:oos 'asdf:load-op x)) '(:elephant :iterate :split-sequence)) (defpackage :nelephant (:use :cl :elephant :iterate :split-sequence )) (in-package :nelephant) (deftype int32 () '(integer 32)) (defconstant +movies-num+ 17770 "Total number of movies") (open-store '(:BDB "/home/victor/netflix/db/")) ;(open-store '(:CLSQL (:SQLITE3 "/home/victor/netflix/sqldb.db"))) (defpclass ratings-collection () ((id :type int32 :initarg :id :reader get-id :index t :documentation "Object id") (ratings :initform (make-btree) :reader ratings :documentation "A hash mapping ids to ratings")) (:documentation "An abstract ratings collection")) (defpclass user (ratings-collection) () (:documentation "Movies rated by user with this ID")) (defpclass movie (ratings-collection) () (:documentation "Users rated movie with this ID")) (defun get-or-create-by-id (id object) "Return existing object with given ID or create a new one if it doesn't exists" (or (first (get-instances-by-value object 'id id)) (make-instance object :id id))) (declaim (inline get-user)) (defun get-user (id) "Return user object with given ID or create a new one" (get-or-create-by-id id 'user)) (declaim (inline get-movie)) (defun get-movie (id) "Return movie object with given ID or create a new one" (get-or-create-by-id id 'movie)) (defun movie-filename (n) "Data file with ratings on film N" (format nil "download/training_set/mv_~7,'0D.txt" n)) (defun collect-rating-info (movie line) "Parse rating data for MOVIE from LINE" (destructuring-bind (user rating date) (split-sequence #\, line) (declare (ignore date)) (setf (get-value movie (ratings (get-user user))) rating (get-value user (ratings (get-movie movie))) rating))) (defun import-movie (n) "Import ratings information for movie number N." (with-transaction () (iter (for line in-file (movie-filename n) using 'read-line) (unless (first-time-p) ; skip first line (collect-rating-info n line))))) (defun fprinc (obj) (princ obj) (princ #\Space) (sb-int:flush-standard-output-streams)) (defun import-all-movies () "Import all netflix movies ratings from text files." (iter (for i from 1 to +movies-num+) (import-movie i) (fprinc i))) On Jan 4, 2008 1:12 AM, Victor Kryukov wrote: > Hello list, > > I've decided to put elephant 0.9.1 under some heavy load test, and > play with Netflix data set a little bit. The attached program that > tries to import everything in BerkeleyDB fails when trying to import > movie file number 8 with the following traceback. Do you have any idea > what could be the problem? > > Thanks, > Victor. > > Berkeley DB error: Cannot allocate memory > [Condition of type BDB-DB-ERROR] > > Restarts: > 0: [ABORT-REQUEST] Abort handling SLIME request. > 1: [ABORT] Exit debugger, returning to top level. > > Backtrace: > 0: ((SB-PCL::FAST-METHOD (SETF GET-VALUE) (T T DB-BDB::BDB-BTREE)) > #) > 1: ((LAMBDA NIL)) > 2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION > (DB-BDB::BDB-STORE-CONTROLLER #1="#<...>" . #1#)) # argument> # # /home/victor/netflix/db/> #) > 3: (IMPORT-ALL-MOVIES) > 4: (SB-INT:SIMPLE-EVAL-IN-LEXENV (IMPORT-ALL-MOVIES) #) > 5: (SWANK::EVAL-REGION "(import-all-movies) > " T) > 6: ((LAMBDA NIL)) > 7: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-SYNTAX-HOOKS (T)) > # # # {100403F959}>) > 8: (SWANK::CALL-WITH-BUFFER-SYNTAX #) > 9: (SB-INT:SIMPLE-EVAL-IN-LEXENV (SWANK:LISTENER-EVAL "(import-all-movies) > ") #) > 10: ((LAMBDA NIL)) > 11: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T > T)) # # # SWANK:SWANK-DEBUGGER-HOOK> #) > 12: ((LAMBDA NIL)) > 13: ((SB-PCL::FAST-METHOD SWANK-BACKEND:CALL-WITH-DEBUGGER-HOOK (T > T)) # # # SWANK:SWANK-DEBUGGER-HOOK> #) > 14: (SWANK::CALL-WITH-REDIRECTED-IO # > #) > 15: (SWANK::CALL-WITH-CONNECTION # > #) > 16: (SWANK::HANDLE-REQUEST #) > 17: (SWANK::PROCESS-AVAILABLE-INPUT # {1002EE6A01}> #) > 18: ((FLET SWANK::HANDLER)) > 19: ((LAMBDA (SWANK-BACKEND::_)) #) > 20: (SB-IMPL::SUB-SERVE-EVENT NIL NIL NIL) > 21: (SB-SYS:WAIT-UNTIL-FD-USABLE 0 :INPUT NIL) > 22: (SB-IMPL::REFILL-INPUT-BUFFER # input" {1002393801}>) > 23: (SB-IMPL::INPUT-CHAR/UTF-8 # input" {1002393801}> NIL #:EOF-OBJECT) > 24: (READ-CHAR # > NIL #:EOF-OBJECT #) > 25: (READ-CHAR # > NIL #:EOF-OBJECT #) > 26: (READ-CHAR # SWANK::*CURRENT-STANDARD-INPUT* {1002384001}> NIL #:EOF-OBJECT > #) > 27: (READ-PRESERVING-WHITESPACE # SWANK::*CURRENT-STANDARD-INPUT* {1002384001}> NIL (NIL) T) > 28: (READ-PRESERVING-WHITESPACE # SWANK::*CURRENT-STANDARD-INPUT* {1002384001}> NIL (NIL) NIL) > 29: (READ # {1002384001}> NIL (NIL) NIL) > 30: (SB-IMPL::REPL-READ-FORM-FUN # SWANK::*CURRENT-STANDARD-INPUT* {1002384001}> #) > 31: (SB-IMPL::REPL-FUN NIL) > 32: (SB-IMPL::REPL-FUN NIL) > 33: ((LAMBDA NIL)) > 34: ((LAMBDA NIL)) > 35: (SB-IMPL::%WITH-REBOUND-IO-SYNTAX #) > 36: (SB-IMPL::TOPLEVEL-REPL NIL) > 37: (SB-IMPL::TOPLEVEL-INIT) > 38: ((LABELS SB-IMPL::RESTART-LISP)) > > > -- > Yours Sincerely, > Victor Kryukov > Chicago GSB class of 2008 > 773-618-9501 > -- Yours Sincerely, Victor Kryukov Chicago GSB class of 2008 773-618-9501 From leslie.polzer at gmx.net Fri Jan 4 08:15:10 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 4 Jan 2008 09:15:10 +0100 (CET) Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: Message-ID: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> > I've decided to put elephant 0.9.1 under some heavy load test, and > play with Netflix data set a little bit. The attached program that > tries to import everything in BerkeleyDB fails when trying to import > movie file number 8 with the following traceback. Do you have any idea > what could be the problem? No, but we might if you tell us - OS - RAM - Swap limits - Kernel limits on memory (e.g. /etc/security/limits.conf for GNU/Linux) - size of the data you're trying to pull Leslie -- My personal blog: http://blog.viridian-project.de/ From victor.kryukov at gmail.com Fri Jan 4 08:41:14 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Fri, 4 Jan 2008 02:41:14 -0600 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> Message-ID: On Jan 4, 2008 2:15 AM, Leslie P. Polzer wrote: > > > I've decided to put elephant 0.9.1 under some heavy load test, and > > play with Netflix data set a little bit. The attached program that > > tries to import everything in BerkeleyDB fails when trying to import > > movie file number 8 with the following traceback. Do you have any idea > > what could be the problem? > > No, but we might if you tell us > > - OS > - RAM > - Swap limits > - Kernel limits on memory (e.g. /etc/security/limits.conf for GNU/Linux) Of course :). # uname -a Linux esculap 2.6.22-14-generic #1 SMP Tue Dec 18 05:28:27 UTC 2007 x86_64 GNU/Linux (Ubuntu 7.10) RAM: Physical 2GB, SWAP 4GB Lisp: SBCL 1.0.13 /etc/security/limits.conf contains only commented out strings. > - size of the data you're trying to pull Well, I'm trying to import Netflix data set. It consists of ratings (from 1 to 5) for 17770 movies given by 480189 different users; total number of ratings is 100480507. Honestly, I don't know how to map that nicely into the Elephant world, so I'm using the following approach; each user (or movies) is a class with 2 slots: id of type (integer 32), and ratings, which is btree mapping movies (or users) ids into respective ratings. I *suspect* that data mapping above is horribly inefficient (too many btrees?), but my intuition may be wrong. Anyway, first 8 files contain exactly 20008 ratings, so at the point of failure we should have created: No more then 8 movie objects No more then 20008 user objects No more then 20016 (= 20008 + 8) btrees (slot elements of the above two classes). I'd be happy to provide more details. Regards, Victor. From eslick at csail.mit.edu Fri Jan 4 08:54:09 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 4 Jan 2008 03:54:09 -0500 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> Message-ID: Hi Victor, Sounds like your transaction is blowing out the shared memory allocated by Berkeley DB to store dirty pages. This is caused by transactions that are too large; putting an entire file of data could well accomplish this. (We really should change the error message to be more informative in these cases). Try pushing with-transaction into the loop in import-movie as follows: (defun import-movie (n) "Import ratings information for movie number N." (iter (for line in-file (movie-filename n) using 'read-line) (unless (first-time-p) ; skip first line (with-transaction () (collect-rating-info n line))))) I expect that will fix it. Ian PS - Your data representation sounds fine as a first start. You may want to rethink it when you start doing queries and in fact may want to build a custom data structure in static memory for performance reasons once your algorithms and related queries are clear to you. PS - You may get some additional mileage by increasing the total memory cache size (you should lookup whether the transaction size is correlated with the overall cache size) using the my-config.sexp option (:berkeley-db-cachesize . 20971520) On Jan 4, 2008, at 3:15 AM, Leslie P. Polzer wrote: > >> I've decided to put elephant 0.9.1 under some heavy load test, and >> play with Netflix data set a little bit. The attached program that >> tries to import everything in BerkeleyDB fails when trying to import >> movie file number 8 with the following traceback. Do you have any >> idea >> what could be the problem? > > No, but we might if you tell us > > - OS > - RAM > - Swap limits > - Kernel limits on memory (e.g. /etc/security/limits.conf for GNU/ > Linux) > - size of the data you're trying to pull > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Fri Jan 4 10:43:17 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 4 Jan 2008 11:43:17 +0100 (CET) Subject: [elephant-devel] Quick sketch of type evolution system In-Reply-To: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> References: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> Message-ID: <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> > to clarify one of the points that are important to me, > I have prepared this working sketch: > > [...] Any reason why no one commented on this? I'd appreciate some feedback. Leslie -- My personal blog: http://blog.viridian-project.de/ From henrik at evahjelte.com Fri Jan 4 12:27:45 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 4 Jan 2008 13:27:45 +0100 Subject: [elephant-devel] Quick sketch of type evolution system In-Reply-To: <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> References: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60801040427n315a3904j7e943bf529c289eb@mail.gmail.com> I am dreaming up something I would like to see, it is perhaps obvious and I haven't thought out all details but anyway: Schemas are stored in the elephant database. Each class has an associated schema class that stores a number of class-versions. Each class-version has a number and a list of slot definitions. Each persistent class instance has a way to identify the schema and actual class-version. For example a class-version-nr, and the schema can be identified from the class-name. (Optional feature: it would be nice if there was a way to rename classes and keep the persistent data, so link the version 1 of my-renamed-class to version 7 of my-old-class) When a class definition is updated, a new class-version is created. New instances get the latest class-version-nr. For each retrieval of an instance, or slot-value of an instance, the version-nr of the instance is checked. If it is outdated, a generic function is called to update it. Something along class-evolve, but I like long names like update-persistent-instance-for-class-redefinition. It is up to the user to keep code to update between versions. If you want to do lazy updates, you will have to keep the old update code in your system for ever. There could be a convenience function called update-all-instances that takes a class-name and does upgrades, after calling it the class-versions objects of older versions may get a known-to-be-obsolete value so you will be able to know that you can drop update code safely. With this schema, you would never actually read from old versions. Every time you query an instance of an old version, the whole instance should be updated slot for slot. So the slot-value-using-class suggested would be unnecessary, there would be no need to upgrade slot types in this function, it should already be done using update-persistent-instance-for-class-redefinition. But, since I personally don't have the time to do this grand plan (now at least), I don't object to the slot-value-using-class updates Leslie proposes. It does not feel like a complete and perfect solution, but won't do no big harm either. But for me, until Elephant gets a complete and proper schema evolution facility I won't trust it to handle lazy updates to slots or objects. So a performance switch to turn it off will be nice. Proper schema handling is perhaps elephants greatest weakness? Best wishes, Henrik Hjelte On Jan 4, 2008 11:43 AM, Leslie P. Polzer wrote: > > > to clarify one of the points that are important to me, > > I have prepared this working sketch: > > > > [...] > > Any reason why no one commented on this? > I'd appreciate some feedback. > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > > -- Henrik Hjelte henrik.hjelte at stix.to +46703993945 http://stix.to L?stmakarg 18-20 (IQube) Box 7438 S-103 91 Stockholm Sweden -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.kryukov at gmail.com Fri Jan 4 08:47:22 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Fri, 4 Jan 2008 02:47:22 -0600 Subject: [elephant-devel] Test failure Message-ID: Hi, Running elephant tests failed (please see attached logs). My configuration: OS: Linux esculap 2.6.22-14-generic #1 SMP Tue Dec 18 05:28:27 UTC 2007 x86_64 GNU/Linux [Ubuntu 7.10] LISP: SBCL 1.0.13 Postgresql: 8.2.5 Elephant: latest version via the darcs RAM: 2GB physical, 4GB swap I've created every Postgresql database required by tests. I'd be happy to help investigate it / provide more info. Best Regards, Victor. -- Yours Sincerely, Victor Kryukov -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: elephant-test-failure.txt URL: From read at robertlread.net Fri Jan 4 16:39:55 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 04 Jan 2008 10:39:55 -0600 Subject: [elephant-devel] Quick sketch of type evolution system In-Reply-To: <50e8e4f60801040427n315a3904j7e943bf529c289eb@mail.gmail.com> References: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> <50e8e4f60801040427n315a3904j7e943bf529c289eb@mail.gmail.com> Message-ID: <1199464795.3740.102.camel@penguin.yourdomain.com> I personally think something like what you are proposing is the direction in which we need to go. I build a schema-revisioning system for a SQL schema system at my previous work; it works along the lines that you suggest, although in LISP it could be a lot better. Of course, there are schema changes which are invisible to the class system, that also need to be accounted for. For example, you represent user status as an integer (representing an enumerated type.) You decide to remove a status. The types haven't changed, but the entire schema has to be massaged into the new set of enumerated types. I think the lack of schema evolution is Elephant's biggest problem; but is there any system that has solved this problem? If you use an ORM, your problems are even worse---you explicitly have to change the object-relational mapping with an class change, and you still have to have a way of updating the system. If the basic problem of schema evolution has been solved by somebody, I would be happy to use their solution as inspiration. Like you, I don't have time to seriously tackle this problem right now---in fact I am still (sigh) debugging the "allocate-instance" patch under postmodern that was submitted, which has very subtle symptoms. On Fri, 2008-01-04 at 13:27 +0100, Henrik Hjelte wrote: > I am dreaming up something I would like to see, it is perhaps obvious > and I haven't thought out all details but anyway: > > Schemas are stored in the elephant database. Each class has an > associated schema class that stores a number of class-versions. Each > class-version has a number and a list of slot definitions. > > Each persistent class instance has a way to identify the schema and > actual class-version. For example a class-version-nr, and the schema > can be identified from the class-name. (Optional feature: it would be > nice if there was a way to rename classes and keep the persistent > data, so link the version 1 of my-renamed-class to version 7 of > my-old-class) > > When a class definition is updated, a new class-version is created. > New instances get the latest class-version-nr. > For each retrieval of an instance, or slot-value of an instance, the > version-nr of the instance is checked. If it is outdated, a generic > function is called to update it. Something along class-evolve, but I > like long names like > update-persistent-instance-for-class-redefinition. It is up to the > user to keep code to update between versions. If you want to do lazy > updates, you will have to keep the old update code in your system for > ever. > > There could be a convenience function called update-all-instances that > takes a class-name and does upgrades, after calling it the > class-versions objects of older versions may get a > known-to-be-obsolete value so > you will be able to know that you can drop update code safely. > > With this schema, you would never actually read from old versions. > Every time you query an instance of an old version, the whole instance > should be updated slot for slot. So the slot-value-using-class > suggested would be unnecessary, there would be no need to upgrade slot > types in this function, it should already be done using > update-persistent-instance-for-class-redefinition. > > But, since I personally don't have the time to do this grand plan (now > at least), I don't object to the slot-value-using-class updates Leslie > proposes. It does not feel like a complete and perfect solution, but > won't do no big harm either. But for me, until Elephant gets a > complete and proper schema evolution facility I won't trust it to > handle lazy updates to slots or objects. So a performance switch to > turn it off will be nice. Proper schema handling is perhaps elephants > greatest weakness? > > Best wishes, > Henrik Hjelte > > > > On Jan 4, 2008 11:43 AM, Leslie P. Polzer > wrote: > > > to clarify one of the points that are important to me, > > I have prepared this working sketch: > > > > > [...] > > Any reason why no one commented on this? > I'd appreciate some feedback. > > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > > > > > > > -- > Henrik Hjelte > henrik.hjelte at stix.to > +46703993945 > http://stix.to > L?stmakarg 18-20 (IQube) > Box 7438 > S-103 91 Stockholm > Sweden > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Fri Jan 4 17:01:01 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 4 Jan 2008 12:01:01 -0500 Subject: [elephant-devel] Quick sketch of type evolution system In-Reply-To: <1199464795.3740.102.camel@penguin.yourdomain.com> References: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> <50e8e4f60801040427n315a3904j7e943bf529c289eb@mail.gmail.com> <1199464795.3740.102.camel@penguin.yourdomain.com> Message-ID: <4C7DCEC1-F5E9-471E-96EF-3473807B0660@csail.mit.edu> I'm dreaming the same dream as Henrik. As I mentioned earlier some of this was captured in the Trak wiki and tickets (I referenced which ones in an earlier e-mail). I'm completely swamped for a couple of weeks, but I might be motivated to work on this later in the month if one or two someone others volunteer to participate. I don't think it's a ton of code, but like most things using the MOP, there are a ton of subtle issues to be thought through. Ian PS - Just to put this idea into the ether: One thought for solving the offline repository problem where I change a schema while the repository is disconnected such that no schema version is created. One approach is to create a signature hash using all the data available in the MOP. When a class object is created by the MOP, it stores this signature. Then, when a repository connects, it verifies that all the latest schemas in the database match the signatures of the classes in memory. If not, it can warn the user or take some other configurable action. update-instance-for-redefined-class would need to call some subsidiary function defined by the user on schema version ids, or signatures, because the generic function dispatch doesn't know about schema versions, just about the prior and the current class objects in memory. On Jan 4, 2008, at 11:39 AM, Robert L. Read wrote: > I personally think something like what you are proposing is the > direction in which we need to go. I build a schema-revisioning system > for a SQL schema system at my previous work; it works along the lines > that you suggest, although in LISP it could be a lot better. > > Of course, there are schema changes which are invisible to the class > system, that also need to be accounted for. For example, you > represent > user status as an integer (representing an enumerated type.) You > decide > to remove a status. The types haven't changed, but the entire schema > has to be massaged into the new set of enumerated types. > > I think the lack of schema evolution is Elephant's biggest problem; > but > is there any system that has solved this problem? If you use an ORM, > your problems are even worse---you explicitly have to change the > object-relational mapping with an class change, and you still have to > have a way of updating the system. If the basic problem of schema > evolution has been solved by somebody, I would be happy to use their > solution as inspiration. > > Like you, I don't have time to seriously tackle this problem right > now---in fact I am still (sigh) debugging the "allocate-instance" > patch > under postmodern that was submitted, which has very subtle symptoms. > > On Fri, 2008-01-04 at 13:27 +0100, Henrik Hjelte wrote: >> I am dreaming up something I would like to see, it is perhaps obvious >> and I haven't thought out all details but anyway: >> >> Schemas are stored in the elephant database. Each class has an >> associated schema class that stores a number of class-versions. Each >> class-version has a number and a list of slot definitions. >> >> Each persistent class instance has a way to identify the schema and >> actual class-version. For example a class-version-nr, and the schema >> can be identified from the class-name. (Optional feature: it would be >> nice if there was a way to rename classes and keep the persistent >> data, so link the version 1 of my-renamed-class to version 7 of >> my-old-class) >> >> When a class definition is updated, a new class-version is created. >> New instances get the latest class-version-nr. >> For each retrieval of an instance, or slot-value of an instance, the >> version-nr of the instance is checked. If it is outdated, a generic >> function is called to update it. Something along class-evolve, but I >> like long names like >> update-persistent-instance-for-class-redefinition. It is up to the >> user to keep code to update between versions. If you want to do lazy >> updates, you will have to keep the old update code in your system for >> ever. >> >> There could be a convenience function called update-all-instances >> that >> takes a class-name and does upgrades, after calling it the >> class-versions objects of older versions may get a >> known-to-be-obsolete value so >> you will be able to know that you can drop update code safely. >> >> With this schema, you would never actually read from old versions. >> Every time you query an instance of an old version, the whole >> instance >> should be updated slot for slot. So the slot-value-using-class >> suggested would be unnecessary, there would be no need to upgrade >> slot >> types in this function, it should already be done using >> update-persistent-instance-for-class-redefinition. >> >> But, since I personally don't have the time to do this grand plan >> (now >> at least), I don't object to the slot-value-using-class updates >> Leslie >> proposes. It does not feel like a complete and perfect solution, but >> won't do no big harm either. But for me, until Elephant gets a >> complete and proper schema evolution facility I won't trust it to >> handle lazy updates to slots or objects. So a performance switch to >> turn it off will be nice. Proper schema handling is perhaps elephants >> greatest weakness? >> >> Best wishes, >> Henrik Hjelte >> >> >> >> On Jan 4, 2008 11:43 AM, Leslie P. Polzer >> wrote: >> >>> to clarify one of the points that are important to me, >>> I have prepared this working sketch: >>> >> >>> [...] >> >> Any reason why no one commented on this? >> I'd appreciate some feedback. >> >> >> Leslie >> >> -- >> My personal blog: http://blog.viridian-project.de/ >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> >> >> >> >> >> -- >> Henrik Hjelte >> henrik.hjelte at stix.to >> +46703993945 >> http://stix.to >> L?stmakarg 18-20 (IQube) >> Box 7438 >> S-103 91 Stockholm >> Sweden >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From luismbo at gmail.com Sat Jan 5 18:42:58 2008 From: luismbo at gmail.com (Luis Oliveira) Date: Sat, 05 Jan 2008 18:42:58 +0000 Subject: [elephant-devel] Trouble compiling Elephant on SBCL Message-ID: Hello, I'm getting a strange error when trying to compile Elephant using SBCL: unknown operator in feature expression: (#:|#.759|). [Condition of type SIMPLE-ERROR] I couldn't pin down where exactly that error is coming from but it seems to happen towards the end of the file because this is the warning that precedes it: ; caught STYLE-WARNING: ; redefining NEW-STYLE-COPY-P in DEFUN Pushing :ELEPHANT-WITHOUT-OPTIMIZE to *FEATURES* made the problem go away, though I'm just starting to use it so all I can say is that OPEN-STORE using CLSQL/SQlite3 works. :-) Here's my system info, in case it turns out to be relevant: $ uname -a Darwin pomajxego 8.11.0 Darwin Kernel Version 8.11.0: Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC Power Macintosh powerpc $ sbcl --version SBCL 1.0.12.15 Also, I'm using the darcs version from a few hours ago: Fri Dec 7 19:54:02 WET 2007 alex.mizrahi at gmail.com * db-postmodern: optimized map-index for -by-value case HTH -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ From read at robertlread.net Sat Jan 5 19:54:37 2008 From: read at robertlread.net (Robert L. Read) Date: Sat, 05 Jan 2008 13:54:37 -0600 Subject: [elephant-devel] Trouble compiling Elephant on SBCL In-Reply-To: References: Message-ID: <1199562877.3740.141.camel@penguin.yourdomain.com> I doubt I will be able to add any insight into this. Can you tell which of the many compiled files it occurs in? I get lots of STYLE-WARINGS under SBCL on Linux; it is very chatty. Some of those things represent things that we could actually improve. On Sat, 2008-01-05 at 18:42 +0000, Luis Oliveira wrote: > Hello, > > I'm getting a strange error when trying to compile Elephant using SBCL: > > unknown operator in feature expression: (#:|#.759|). > [Condition of type SIMPLE-ERROR] > > I couldn't pin down where exactly that error is coming from but it seems > to happen towards the end of the file because this is the warning that > precedes it: > > ; caught STYLE-WARNING: > ; redefining NEW-STYLE-COPY-P in DEFUN > > > Pushing :ELEPHANT-WITHOUT-OPTIMIZE to *FEATURES* made the problem go > away, though I'm just starting to use it so all I can say is that > OPEN-STORE using CLSQL/SQlite3 works. :-) Here's my system info, in case > it turns out to be relevant: > > $ uname -a > Darwin pomajxego 8.11.0 Darwin Kernel Version 8.11.0: > Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC > Power Macintosh powerpc > > $ sbcl --version > SBCL 1.0.12.15 > > Also, I'm using the darcs version from a few hours ago: > > Fri Dec 7 19:54:02 WET 2007 alex.mizrahi at gmail.com > * db-postmodern: optimized map-index for -by-value case > > HTH > From eslick at csail.mit.edu Sat Jan 5 20:04:51 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 5 Jan 2008 15:04:51 -0500 Subject: [elephant-devel] Trouble compiling Elephant on SBCL In-Reply-To: References: Message-ID: <652C6C0B-2CBB-46C2-9D93-C95BFE99FE9F@csail.mit.edu> We'll need alot more information to help. Can you tell us on what file the compile failed, and provide the complete compilation output prior to that point? Also you say that you can't compile elephant, but that open-store for CLSQL works? Do you mean you have a problem in a specific backend such as BDB? Ian On Jan 5, 2008, at 1:42 PM, Luis Oliveira wrote: > Hello, > > I'm getting a strange error when trying to compile Elephant using > SBCL: > > unknown operator in feature expression: (#:|#.759|). > [Condition of type SIMPLE-ERROR] > > I couldn't pin down where exactly that error is coming from but it > seems > to happen towards the end of the file because this is the warning that > precedes it: > > ; caught STYLE-WARNING: > ; redefining NEW-STYLE-COPY-P in DEFUN > > > Pushing :ELEPHANT-WITHOUT-OPTIMIZE to *FEATURES* made the problem go > away, though I'm just starting to use it so all I can say is that > OPEN-STORE using CLSQL/SQlite3 works. :-) Here's my system info, in > case > it turns out to be relevant: > > $ uname -a > Darwin pomajxego 8.11.0 Darwin Kernel Version 8.11.0: > Wed Oct 10 18:26:00 PDT 2007; root:xnu-792.24.17~1/RELEASE_PPC > Power Macintosh powerpc > > $ sbcl --version > SBCL 1.0.12.15 > > Also, I'm using the darcs version from a few hours ago: > > Fri Dec 7 19:54:02 WET 2007 alex.mizrahi at gmail.com > * db-postmodern: optimized map-index for -by-value case > > HTH > > -- > Lu?s Oliveira > http://student.dei.uc.pt/~lmoliv/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Sat Jan 5 21:58:36 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 5 Jan 2008 23:58:36 +0200 Subject: [elephant-devel] Re: Trouble compiling Elephant on SBCL References: <652C6C0B-2CBB-46C2-9D93-C95BFE99FE9F@csail.mit.edu> Message-ID: IE> We'll need alot more information to help. my telepathy says it's stuff in memutil.lisp ;; Thanks to Juho Snellman for this idiom. (eval-when (:compile-toplevel) (defun new-style-copy-p () #+(and sbcl sb-unicode) (if (find-symbol "COPY-UB8-FROM-SYSTEM-AREA" "SB-KERNEL") '(:and) '(:or)) #-(and sbcl sb-unicode) t)) #+(and sbcl sb-unicode) (let ((res (make-string byte-length :element-type 'base-char))) #+#.(elephant-memutil::new-style-copy-p) (sb-kernel:copy-ub8-from-system-area (sb-alien:alien-sap (buffer-stream-buffer bs)) position res 0 byte-length) it seems #+#. trick got somehow broken with new version of SBCL with optimization settings. comment says ;; This code is an attempt to allow compilation under bothe SBCL 8 and SBCL 9. so perhaps this stuff ain't needed for now. From victor.kryukov at gmail.com Sun Jan 6 00:02:03 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Sat, 5 Jan 2008 18:02:03 -0600 Subject: [elephant-devel] Berkeley DB error: Cannot allocate memory. In-Reply-To: References: <64662.88.73.241.96.1199434510.squirrel@mail.stardawn.org> Message-ID: On Jan 4, 2008 2:54 AM, Ian Eslick wrote: > Hi Victor, > > Sounds like your transaction is blowing out the shared memory > allocated by Berkeley DB to store dirty pages. This is caused by > transactions that are too large; putting an entire file of data could > well accomplish this. (We really should change the error message to > be more informative in these cases). > > Try pushing with-transaction into the loop in import-movie as follows: Thanks for your suggestion, Ian - the problem was solved once I've moved with-transaction inside the collect-rating-info. From leslie.polzer at gmx.net Sun Jan 6 15:01:09 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 6 Jan 2008 16:01:09 +0100 (CET) Subject: Schema evolution (was: Re: [elephant-devel] Quick sketch of type evolution system) In-Reply-To: <4C7DCEC1-F5E9-471E-96EF-3473807B0660@csail.mit.edu> References: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> <50e8e4f60801040427n315a3904j7e943bf529c289eb@mail.gmail.com> <1199464795.3740.102.camel@penguin.yourdomain.com> <4C7DCEC1-F5E9-471E-96EF-3473807B0660@csail.mit.edu> Message-ID: <64587.88.73.215.66.1199631669.squirrel@mail.stardawn.org> > I'm dreaming the same dream as Henrik. As I mentioned earlier some of > this was captured in the Trak wiki and tickets (I referenced which > ones in an earlier e-mail). They were very valuable to me in helping me understanding the issues and questions involved. > I'm completely swamped for a couple of weeks, but I might be motivated > to work on this later in the month if one or two someone others > volunteer to participate. I volunteer for helping to the best I'm able to. > One approach is to create a signature hash [...] Fair enough. We need to find a balance between storing schema versions and letting the user decide. What I'd like to see is 1) easy userland specification of schema versions and transitions between those versions (from older to newer should suffice) 2) storage of at least one schema version in the repo; this would, for example, make a PHPMyAdmin-style web interface possible, i.e. some piece of software that can be used on any Elephant repo without access to the user code that created it. Leslie -- My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Sun Jan 6 17:14:00 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 6 Jan 2008 18:14:00 +0100 (CET) Subject: [elephant-devel] Quick sketch of type evolution system In-Reply-To: <1199464795.3740.102.camel@penguin.yourdomain.com> References: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> <50e8e4f60801040427n315a3904j7e943bf529c289eb@mail.gmail.com> <1199464795.3740.102.camel@penguin.yourdomain.com> Message-ID: <64475.88.73.215.66.1199639640.squirrel@mail.stardawn.org> > Of course, there are schema changes which are invisible to the class > system, that also need to be accounted for. For example, you represent > user status as an integer (representing an enumerated type.) You decide > to remove a status. The types haven't changed, but the entire schema > has to be massaged into the new set of enumerated types. This would be the responsbility of the userland part: let the programmer bump the schema's version number to indicate a change and then provide a transition function. > I think the lack of schema evolution is Elephant's biggest problem; but > is there any system that has solved this problem? Let's be the first one. Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at csail.mit.edu Sun Jan 6 17:29:26 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 6 Jan 2008 12:29:26 -0500 Subject: [elephant-devel] Quick sketch of type evolution system In-Reply-To: <64475.88.73.215.66.1199639640.squirrel@mail.stardawn.org> References: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> <63253.88.73.241.96.1199443397.squirrel@mail.stardawn.org> <50e8e4f60801040427n315a3904j7e943bf529c289eb@mail.gmail.com> <1199464795.3740.102.camel@penguin.yourdomain.com> <64475.88.73.215.66.1199639640.squirrel@mail.stardawn.org> Message-ID: <15F606D4-0D79-4516-91A8-E27CA3C25780@csail.mit.edu> That sounds like a good engineering tradeoff. Perhaps we can write a condition check that warns users that they're altering a schema which is would result in incompatible types, dropped slots, and other changes that may require a schema evolution function and bumping the schema #. We could also warn if there is not a database connected. We should also have a warning if a redefinition is causing the system to drop the class index. (And a method for walking the DB and constructing a class index from all reachable instances) For this scheme to work, however, I think we need to think carefully about whether we want to support multiple data stores. We don't want the system to quietly screw up. If we provide an API it should do the 'right thing' or warn the user. I think this is harder with multiple stores. I think we probably should only provide guarantees when there is a 1:1 correspondence between class and store; if you want to use a class in multiple stores we'll provide some documentation but you're on your own. (i.e. we have to figure out in which stores we have to update schema objects and it is easy to have it updated in one that was connected, but not in one that wasn't connected, etc) Ian PS - I'm going to be effectively offline until the 15th so will pick up on the thread of this conversation then. On Jan 6, 2008, at 12:14 PM, Leslie P. Polzer wrote: > >> Of course, there are schema changes which are invisible to the class >> system, that also need to be accounted for. For example, you >> represent >> user status as an integer (representing an enumerated type.) You >> decide >> to remove a status. The types haven't changed, but the entire schema >> has to be massaged into the new set of enumerated types. > > This would be the responsbility of the userland part: let the > programmer bump the > schema's version number to indicate a change and then provide a > transition function. > > >> I think the lack of schema evolution is Elephant's biggest problem; >> but >> is there any system that has solved this problem? > > Let's be the first one. > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From victor.kryukov at gmail.com Mon Jan 7 07:42:47 2008 From: victor.kryukov at gmail.com (Victor Kryukov) Date: Mon, 7 Jan 2008 01:42:47 -0600 Subject: [elephant-devel] String or BLOB exceeded size limit Message-ID: Hello list, while working with sqlite backend, I sometimes get the following error message: Error 18 / String or BLOB exceeded size limit has occurred. [Condition of type CLSQL-SYS:SQL-DATABASE-DATA-ERROR] It happens when applications tries to serialize object which is too big to fit into SQLITE blob or string. Do you think this should be considered as a (non critical) defect[1], or it's not important at all because everybody is using Postgres / Berkeley DB for the backend? [1] ... and elephant should catch this error tryies to split object serialization into smaller pieces Regards, -- Yours Sincerely, Victor Kryukov From killerstorm at newmail.ru Mon Jan 7 13:37:55 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 7 Jan 2008 15:37:55 +0200 Subject: [elephant-devel] optimized map-index? Message-ID: helo i've implemented optimized map-index specialization to get better performance for get-instances-by-value. but for now "better cursors" i've implemented later and Ian's optimized map-index implementations should do pretty well too. numerically i have following stats for doing 100 get-instances-by-value queries on 10000 elements table: normal map-index: 0.284 seconds of real time 0.032002 seconds of user run time 0.136009 seconds of system run time 0 calls to %EVAL 0 page faults and 3,289,600 bytes consed. specialized map-index: Evaluation took: 0.221 seconds of real time 0.004001 seconds of user run time 0.096006 seconds of system run time 0 calls to %EVAL 0 page faults and 1,555,536 bytes consed. it appears walking by cursors introduce some overhead (twice more consing), but it's not that significant comparing to overall communication overhead. on PostgreSQL side, queries made by cursor are more complex, but again difference is not significant: something like 0.03 msecs for direct and 0.04 for cursor ones. so, i'm not sure if i should delete specialized map-index version. benefits of pure cursor version: * no additional code to maintain, less complexity * works in iterative way: if we have *lots* of instances with specific value, cursor version will allow to iterate through them one at time (well, 10 at time), and it's possible to abort iteration. (other version will try to fit everything in memory or will fail) benefits of optimized version: * works somewhat faster * uses straightforward SQL query thinking........ seems "pure" version benefits outweight optimized one, so i'm deleting specialization.. From kazennikov.nntp at gmail.com Tue Jan 8 09:10:26 2008 From: kazennikov.nntp at gmail.com (Anton Kazennikov) Date: Tue, 08 Jan 2008 12:10:26 +0300 Subject: [elephant-devel] Re: Trouble compiling Elephant on SBCL References: <652C6C0B-2CBB-46C2-9D93-C95BFE99FE9F@csail.mit.edu> Message-ID: <87k5mkhed9.fsf@kzn.homelinux.org> "Alex Mizrahi" writes: > IE> We'll need alot more information to help. > > my telepathy says it's stuff in memutil.lisp > > it seems #+#. trick got somehow broken with new version of SBCL with > optimization settings. > Hmm.. I've got this problem when compiling under slime with load-system. When I do this from prompt using (asdf:oos 'asdf:load-op :elephant) the problem is gone. Such strange heuristic ;-) -- With best regards, Anton Kazennikov. mailto:kazennikov[at]gmail.com ICQ# 98965967 From killerstorm at newmail.ru Tue Jan 8 12:10:17 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 8 Jan 2008 14:10:17 +0200 Subject: [elephant-devel] recreate-instance-using-class Message-ID: helo there was a patch that alters the way how objects that are deserialized are created: it uses allocate-instance and bypasses normal initialization sequence of make-instance. however, there was no documentation given how this is supposed to work, so i thought this shouldn't affect applications. but we've found that sometimes it has disastrous effects in some cases. for example, we've found that "strange bug" Robert saw in postmodern backend happens because initialize-instance of pm-btree is not called. it seems now we should use recreate-instance instead of initialize-instance, because descendants of "persistent", like btrees and other internal classes, are completely deprived from normal Common Lisp initialization functions. if this is intentional, probably it's worth documenting this, because finding such stuff from weird bugs isn't very pleasant. also, it seems initargs/initforms won't be initialized on recreated instances of persistent, at least i don't see any way how they could be initialized. we should forget about this functionality for internal elephant's persistent classes? or this damage was not intentional? as i understand, elephant users are supposed to work with persistent-object, but not persistent class, so maybe this patch should only affect persistent-object? it's also quite strange that recreate-instance for persistent-object calls shared-initialize, but for persistent it doesn't. looks like intentional sabotage! :) but it's not clear how this stuff should affect descendants of persistent-object. if people used initialize-instance :after to intialize transient slots, how are they supposed to intialize them now? shared-initialize :after? or they should use ele::recreate-instance? with best regards, Alex 'killer_storm' Mizrahi. From henrik at evahjelte.com Tue Jan 8 13:39:08 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Tue, 8 Jan 2008 14:39:08 +0100 Subject: [elephant-devel] Some patches Message-ID: <50e8e4f60801080539t3563ccf8qf572a4509499de8d@mail.gmail.com> Here are some patches that are necessary for db-postmodern. Also: If you use the postmodern backend it is currently incompatible with the last two patches from sross, so unpull them for little while until we have discussed how they should work. Best wishes, Henrik -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: patches.darcs Type: application/octet-stream Size: 52668 bytes Desc: not available URL: From eslick at csail.mit.edu Tue Jan 8 13:50:33 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 8 Jan 2008 08:50:33 -0500 Subject: [elephant-devel] recreate-instance-using-class In-Reply-To: References: Message-ID: Hi Alex, I don't think any of the current developers have sat down and really thought through how the entire MOP interface _should_ work in light of our learning. My guess is that getting all this right can best be accomplished by rationally designing all the functionality from scratch, and rewrite the persistence protocol as necessary to accommodate the new design. I learned the MOP during my rewrite a year or two ago, so I'm sure there is some evidence of this in the current implementation. With the expanded test suite, debugging the new MOP implementation shouldn't be overwhelmingly problematic. There are several orthogonal requirements being satisfied during initialization that should be separated clearly in the design: 1) CLOS instance mgmt (such as allocate-instance) 2) Elephant instance mgmt (all elephant instances need an OID, a home store, etc) 3) CLOS/Elephant slot value initialization a) During creation (all arguments, initforms; write db) b) During deserialization (initforms for transient slots only) 4) Slot value access i) For now, always directly to DB ii) maintain indexes Every time a persistent object is created or recreated, steps #1 and #2 are required. #1 and #2 are all tied up in shared-initialize and initialize-instance, and I don't think there is a clean separation between those steps and the steps in #3 that handle the initialization vs. re-initialization problems. Additional complications: - Schema evolution makes this even more exciting and requires intervention in elephant instance management, slot value initialization and perhaps even slot value access. - An additional complication has been dealing with variations among different MOP implementations. - We've talked about having a clean way to allow slots to be cached, i.e. to be declared unshared so they only access the DB on writes, or managed which means only written when 'saved' Ian On Jan 8, 2008, at 7:10 AM, Alex Mizrahi wrote: > helo > > there was a patch that alters the way how objects that are > deserialized are > created: it uses allocate-instance and bypasses normal initialization > sequence of make-instance. > > however, there was no documentation given how this is supposed to > work, so i > thought this shouldn't affect applications. > > but we've found that sometimes it has disastrous effects in some > cases. > > for example, we've found that "strange bug" Robert saw in postmodern > backend > happens because initialize-instance of pm-btree is not called. > > it seems now we should use recreate-instance instead of initialize- > instance, > because descendants of "persistent", like btrees and other internal > classes, > are completely deprived from normal Common Lisp initialization > functions. if > this is intentional, probably it's worth documenting this, because > finding > such stuff from weird bugs isn't very pleasant. > > also, it seems initargs/initforms won't be initialized on recreated > instances of persistent, at least i don't see any way how they could > be > initialized. we should forget about this functionality for internal > elephant's persistent classes? > > or this damage was not intentional? as i understand, elephant users > are > supposed to work with persistent-object, but not persistent class, > so maybe > this patch should only affect persistent-object? > > it's also quite strange that recreate-instance for persistent-object > calls > shared-initialize, but for persistent it doesn't. looks like > intentional > sabotage! :) > > but it's not clear how this stuff should affect descendants of > persistent-object. if people used initialize-instance :after to > intialize > transient slots, how are they supposed to intialize them now? > shared-initialize :after? or they should use ele::recreate-instance? > > with best regards, Alex 'killer_storm' Mizrahi. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Tue Jan 8 14:41:35 2008 From: read at robertlread.net (Robert L. Read) Date: Tue, 08 Jan 2008 08:41:35 -0600 Subject: [elephant-devel] recreate-instance-using-class In-Reply-To: References: Message-ID: <1199803295.3740.209.camel@penguin.yourdomain.com> I agree completely with Ian. In fact I have now spent a very embarrassing 5 days trying to modify the ele-postmodern interface to use "recreate-instance" based on Sean's patches, and have failed miserably. This is the first time in my career this has ever happened to me. I wasted this time in part to learn about postmodern, and in part because I didn't want to reject Sean's patch, which apparently he and one other person need. However, it seems clear now that we need to back Sean's patch out of the repository and answer more of the questions that Ian raises. In particular, it was never perfectly clear to me why Sean thought avoiding the call to "make-instance" was so important. I understand of course that if one has overloaded make-instance with some functionality that has nothing to do with reconstituting the object, the current implementation of Elephant will erroneously invoke that; but it is not clear if Elephant should change to suit that, or if the user should simply move that functionality out of make-instance. That is part of the policy question we will have to answer. On Tue, 2008-01-08 at 08:50 -0500, Ian Eslick wrote: > Hi Alex, > > I don't think any of the current developers have sat down and really > thought through how the entire MOP interface _should_ work in light of > our learning. My guess is that getting all this right can best be > accomplished by rationally designing all the functionality from > scratch, and rewrite the persistence protocol as necessary to > accommodate the new design. I learned the MOP during my rewrite a > year or two ago, so I'm sure there is some evidence of this in the > current implementation. With the expanded test suite, debugging the > new MOP implementation shouldn't be overwhelmingly problematic. > > There are several orthogonal requirements being satisfied during > initialization that should be separated clearly in the design: > > 1) CLOS instance mgmt (such as allocate-instance) > 2) Elephant instance mgmt (all elephant instances need an OID, a home > store, etc) > 3) CLOS/Elephant slot value initialization > a) During creation (all arguments, initforms; write db) > b) During deserialization (initforms for transient slots only) > 4) Slot value access > i) For now, always directly to DB > ii) maintain indexes > > Every time a persistent object is created or recreated, steps #1 and > #2 are required. #1 and #2 are all tied up in shared-initialize and > initialize-instance, and I don't think there is a clean separation > between those steps and the steps in #3 that handle the initialization > vs. re-initialization problems. > > Additional complications: > - Schema evolution makes this even more exciting and requires > intervention in elephant instance management, slot value > initialization and perhaps even slot value access. > - An additional complication has been dealing with variations among > different MOP implementations. > - We've talked about having a clean way to allow slots to be cached, > i.e. to be declared unshared so they only access the DB on writes, or > managed which means only written when 'saved' > > Ian > > > On Jan 8, 2008, at 7:10 AM, Alex Mizrahi wrote: > > > helo > > > > there was a patch that alters the way how objects that are > > deserialized are > > created: it uses allocate-instance and bypasses normal initialization > > sequence of make-instance. > > > > however, there was no documentation given how this is supposed to > > work, so i > > thought this shouldn't affect applications. > > > > but we've found that sometimes it has disastrous effects in some > > cases. > > > > for example, we've found that "strange bug" Robert saw in postmodern > > backend > > happens because initialize-instance of pm-btree is not called. > > > > it seems now we should use recreate-instance instead of initialize- > > instance, > > because descendants of "persistent", like btrees and other internal > > classes, > > are completely deprived from normal Common Lisp initialization > > functions. if > > this is intentional, probably it's worth documenting this, because > > finding > > such stuff from weird bugs isn't very pleasant. > > > > also, it seems initargs/initforms won't be initialized on recreated > > instances of persistent, at least i don't see any way how they could > > be > > initialized. we should forget about this functionality for internal > > elephant's persistent classes? > > > > or this damage was not intentional? as i understand, elephant users > > are > > supposed to work with persistent-object, but not persistent class, > > so maybe > > this patch should only affect persistent-object? > > > > it's also quite strange that recreate-instance for persistent-object > > calls > > shared-initialize, but for persistent it doesn't. looks like > > intentional > > sabotage! :) > > > > but it's not clear how this stuff should affect descendants of > > persistent-object. if people used initialize-instance :after to > > intialize > > transient slots, how are they supposed to intialize them now? > > shared-initialize :after? or they should use ele::recreate-instance? > > > > with best regards, Alex 'killer_storm' Mizrahi. > > > > > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Tue Jan 8 14:47:58 2008 From: read at robertlread.net (Robert L. Read) Date: Tue, 08 Jan 2008 08:47:58 -0600 Subject: [elephant-devel] I've rolledback Sean Ross's initialization patch... In-Reply-To: <1199803295.3740.209.camel@penguin.yourdomain.com> References: <1199803295.3740.209.camel@penguin.yourdomain.com> Message-ID: <1199803678.3740.211.camel@penguin.yourdomain.com> Until we can resolve the issues Ian described, or at least make it work with ele-postmodern. From eslick at csail.mit.edu Tue Jan 8 15:31:13 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 8 Jan 2008 10:31:13 -0500 Subject: [elephant-devel] recreate-instance-using-class In-Reply-To: <1199803295.3740.209.camel@penguin.yourdomain.com> References: <1199803295.3740.209.camel@penguin.yourdomain.com> Message-ID: <75FD8FA7-E4B7-42E6-9E5C-CD3838E1387B@csail.mit.edu> Actually I think we definitely need to fix this. Quite a few people have run into the problem that we're violating the CLOS contract on make-instance and that we should not use the same CLOS calling path for re-initialization that we use for initial creation. I think we should require new contracts rather than messing with familiar ones. One proposal was to create a placeholder class and then call change- class on it, but that still evokes the initargs for new slots and creates problems of its own. Another one is to make a cheap placeholder that is only initialized when touched, which I like but can't think of how to implement. As the next step, I think we probably want to figure out, for deserialized instances, how to create a minimal instance, initializing transient slots only, handle any schema evolution we choose to implement and then call a generic function (reconstitute-instance) that users can define methods on to do any deserialization time specialization. This way the users can do the usual 'on creation' specialization on initialize-instance without having it called multiple times in unexpected ways. The other benefit is that we do less total computation when we load instances into memory, especially when we are unlikely to access them. The trick is to initialize the instance and transient slots without calling all the make-instance, initialize-instance machinery. I'd have to go back and get my head around the MOP details again to suggest the best way to do this (parallel code, checks in initialize- instance, etc). The current initialization code is actually pretty ugly which is undoubtably why there are so many problems trying to patch it, so it could really do with a clean rewrite. Ian On Jan 8, 2008, at 9:41 AM, Robert L. Read wrote: > I agree completely with Ian. > > In fact I have now spent a very embarrassing 5 days trying to modify > the > ele-postmodern interface to use "recreate-instance" based on Sean's > patches, and have failed miserably. This is the first time in my > career > this has ever happened to me. > > I wasted this time in part to learn about postmodern, and in part > because I didn't want to reject Sean's patch, which apparently he and > one other person need. However, it seems clear now that we need to > back > Sean's patch out of the repository and answer more of the questions > that > Ian raises. In particular, it was never perfectly clear to me why > Sean > thought avoiding the call to "make-instance" was so important. I > understand of course that if one has overloaded make-instance with > some > functionality that has nothing to do with reconstituting the object, > the > current implementation of Elephant will erroneously invoke that; but > it > is not clear if Elephant should change to suit that, or if the user > should simply move that functionality out of make-instance. That is > part of the policy question we will have to answer. > > On Tue, 2008-01-08 at 08:50 -0500, Ian Eslick wrote: >> Hi Alex, >> >> I don't think any of the current developers have sat down and really >> thought through how the entire MOP interface _should_ work in light >> of >> our learning. My guess is that getting all this right can best be >> accomplished by rationally designing all the functionality from >> scratch, and rewrite the persistence protocol as necessary to >> accommodate the new design. I learned the MOP during my rewrite a >> year or two ago, so I'm sure there is some evidence of this in the >> current implementation. With the expanded test suite, debugging the >> new MOP implementation shouldn't be overwhelmingly problematic. >> >> There are several orthogonal requirements being satisfied during >> initialization that should be separated clearly in the design: >> >> 1) CLOS instance mgmt (such as allocate-instance) >> 2) Elephant instance mgmt (all elephant instances need an OID, a home >> store, etc) >> 3) CLOS/Elephant slot value initialization >> a) During creation (all arguments, initforms; write db) >> b) During deserialization (initforms for transient slots only) >> 4) Slot value access >> i) For now, always directly to DB >> ii) maintain indexes >> >> Every time a persistent object is created or recreated, steps #1 and >> #2 are required. #1 and #2 are all tied up in shared-initialize and >> initialize-instance, and I don't think there is a clean separation >> between those steps and the steps in #3 that handle the >> initialization >> vs. re-initialization problems. >> >> Additional complications: >> - Schema evolution makes this even more exciting and requires >> intervention in elephant instance management, slot value >> initialization and perhaps even slot value access. >> - An additional complication has been dealing with variations among >> different MOP implementations. >> - We've talked about having a clean way to allow slots to be cached, >> i.e. to be declared unshared so they only access the DB on writes, or >> managed which means only written when 'saved' >> >> Ian >> >> >> On Jan 8, 2008, at 7:10 AM, Alex Mizrahi wrote: >> >>> helo >>> >>> there was a patch that alters the way how objects that are >>> deserialized are >>> created: it uses allocate-instance and bypasses normal >>> initialization >>> sequence of make-instance. >>> >>> however, there was no documentation given how this is supposed to >>> work, so i >>> thought this shouldn't affect applications. >>> >>> but we've found that sometimes it has disastrous effects in some >>> cases. >>> >>> for example, we've found that "strange bug" Robert saw in postmodern >>> backend >>> happens because initialize-instance of pm-btree is not called. >>> >>> it seems now we should use recreate-instance instead of initialize- >>> instance, >>> because descendants of "persistent", like btrees and other internal >>> classes, >>> are completely deprived from normal Common Lisp initialization >>> functions. if >>> this is intentional, probably it's worth documenting this, because >>> finding >>> such stuff from weird bugs isn't very pleasant. >>> >>> also, it seems initargs/initforms won't be initialized on recreated >>> instances of persistent, at least i don't see any way how they could >>> be >>> initialized. we should forget about this functionality for internal >>> elephant's persistent classes? >>> >>> or this damage was not intentional? as i understand, elephant users >>> are >>> supposed to work with persistent-object, but not persistent class, >>> so maybe >>> this patch should only affect persistent-object? >>> >>> it's also quite strange that recreate-instance for persistent-object >>> calls >>> shared-initialize, but for persistent it doesn't. looks like >>> intentional >>> sabotage! :) >>> >>> but it's not clear how this stuff should affect descendants of >>> persistent-object. if people used initialize-instance :after to >>> intialize >>> transient slots, how are they supposed to intialize them now? >>> shared-initialize :after? or they should use ele::recreate-instance? >>> >>> with best regards, Alex 'killer_storm' Mizrahi. >>> >>> >>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Tue Jan 8 16:08:11 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 8 Jan 2008 18:08:11 +0200 Subject: [elephant-devel] Re: recreate-instance-using-class References: Message-ID: IE> our learning. My guess is that getting all this right can best be IE> accomplished by rationally designing all the functionality from IE> scratch, and rewrite the persistence protocol as necessary to IE> accommodate the new design. the way it was working before "the patches" seemed to be more-or-less consistent to me. at least it wasn't that confusing.. initialize-instance wasn't very pretty, but at least it was possible to make it working as it should. it seems reasonable to me that users who need distinguish initial creation from re-creation can check this in their code and act accordingly. separate generic function for transient initialization might look cleaner, but it can seem more complicated to users.. From read at robertlread.net Tue Jan 8 16:15:29 2008 From: read at robertlread.net (Robert L. Read) Date: Tue, 08 Jan 2008 10:15:29 -0600 Subject: [elephant-devel] recreate-instance-using-class In-Reply-To: <75FD8FA7-E4B7-42E6-9E5C-CD3838E1387B@csail.mit.edu> References: <1199803295.3740.209.camel@penguin.yourdomain.com> <75FD8FA7-E4B7-42E6-9E5C-CD3838E1387B@csail.mit.edu> Message-ID: <1199808929.3783.1.camel@penguin.yourdomain.com> On Tue, 2008-01-08 at 10:31 -0500, Ian Eslick wrote: > Actually I think we definitely need to fix this. Quite a few people > have run into the problem that we're violating the CLOS contract on > make-instance and that we should not use the same CLOS calling path > for re-initialization that we use for initial creation. I think we > should require new contracts rather than messing with familiar ones. Is this in the CLHS? Where is this contract documented? The contract is that make-instance should only be called once for a given object? From eslick at csail.mit.edu Tue Jan 8 16:24:08 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 8 Jan 2008 11:24:08 -0500 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: References: Message-ID: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> On Jan 8, 2008, at 11:08 AM, Alex Mizrahi wrote: > IE> our learning. My guess is that getting all this right can best be > IE> accomplished by rationally designing all the functionality from > IE> scratch, and rewrite the persistence protocol as necessary to > IE> accommodate the new design. > > the way it was working before "the patches" seemed to be more-or-less > consistent to me. > at least it wasn't that confusing.. I think the problem is that initialize-instance gets called each time, so if you define an :after method on it for object creation time - for example that computes a specific slot value, then that code gets rerun on object re-initialization and clobbers any updates that happened in the meantime. Please correct me folks if that isn't the experience or there are other issues too. > initialize-instance wasn't very pretty, but at least it was possible > to make > it working as it should. > it seems reasonable to me that users who need distinguish initial > creation > from re-creation can check this in their code and act accordingly. > > separate generic function for transient initialization might look > cleaner, > but it can seem more complicated to users.. Actually this would happen as expected. Objects loaded into memory would automatically get the initforms of the transient fields, unless they wanted to override the default by calling the special generic function. > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Tue Jan 8 16:54:13 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 8 Jan 2008 18:54:13 +0200 Subject: [elephant-devel] Re: recreate-instance-using-class References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> Message-ID: IE>>> our learning. My guess is that getting all this right can best be IE>>> accomplished by rationally designing all the functionality from IE>>> scratch, and rewrite the persistence protocol as necessary to IE>>> accommodate the new design. ??>> ??>> the way it was working before "the patches" seemed to be more-or-less ??>> consistent to me. ??>> at least it wasn't that confusing.. IE> I think the problem is that initialize-instance gets called each time, IE> so if you define an :after method on it for object creation time - for IE> example that computes a specific slot value, then that code gets rerun IE> on object re-initialization and clobbers any updates that happened in IE> the meantime. what's wrong with having such structure of initialize-instance: (defmethod initialize-instance :after ((instance some-class) &rest initargs &allow-other-keys) (unless (member :from-oid initargs) ;; modify persistent slots for the first time ) ;; do "transient" initialization here ) ? IE> Actually this would happen as expected. Objects loaded into memory IE> would automatically get the initforms of the transient fields, unless IE> they wanted to override the default by calling the special generic IE> function. i thought people might be doing all sorts of funny stuff in their initialize-instances -- additional initialization unrelated to instance slots, validation, writing to logs etc. and they can be surprised that once they make class elephant-persistent and it's loaded from DB, their stuff is not executed anymore. From eslick at csail.mit.edu Tue Jan 8 17:38:20 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 8 Jan 2008 12:38:20 -0500 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> Message-ID: <4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> > > i thought people might be doing all sorts of funny stuff in their > initialize-instances -- additional initialization unrelated to > instance > slots, validation, writing to logs etc. > and they can be surprised that once they make class elephant- > persistent and > it's loaded from DB, their stuff is not executed anymore. > Isn't is the other way around? Wouldn't they be surprised when they get a log entry for user creation, or send a confirmation e-mail, not just for when a user was added, but for every time a cursor touched that object by pulling it out of the database? From rosssd at gmail.com Tue Jan 8 17:45:56 2008 From: rosssd at gmail.com (Sean Ross) Date: Tue, 8 Jan 2008 17:45:56 +0000 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: <4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> <4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> Message-ID: <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> Hi all, I'm sorry to hear that my patch caused so many issues with postmodern and I'm quite keen to investigate the cause of this. I'll be installing setting up the postmodern backend and seeing if I can track down the causes of these problems. As to where to go from here I do agree with robert on this and that it is nice to have the seperation between the creation and restoration of instances although I think this needs to be done with a well thought out protocol. cheers, sean. From rosssd at gmail.com Tue Jan 8 19:31:10 2008 From: rosssd at gmail.com (Sean Ross) Date: Tue, 8 Jan 2008 19:31:10 +0000 Subject: [elephant-devel] recreate-instance-using-class In-Reply-To: <75FD8FA7-E4B7-42E6-9E5C-CD3838E1387B@csail.mit.edu> References: <1199803295.3740.209.camel@penguin.yourdomain.com> <75FD8FA7-E4B7-42E6-9E5C-CD3838E1387B@csail.mit.edu> Message-ID: <5bef28df0801081131o28a0f35and29c5948d11e64b4@mail.gmail.com> On Jan 8, 2008 3:31 PM, Ian Eslick wrote: > Actually I think we definitely need to fix this. Quite a few people > have run into the problem that we're violating the CLOS contract on > make-instance and that we should not use the same CLOS calling path > for re-initialization that we use for initial creation. I think we > should require new contracts rather than messing with familiar ones. > > One proposal was to create a placeholder class and then call change- > class on it, but that still evokes the initargs for new slots and > creates problems of its own. Another one is to make a cheap > placeholder that is only initialized when touched, which I like but > can't think of how to implement. > > As the next step, I think we probably want to figure out, for > deserialized instances, how to create a minimal instance, initializing > transient slots only, handle any schema evolution we choose to > implement and then call a generic function (reconstitute-instance) > that users can define methods on to do any deserialization time > specialization. This way the users can do the usual 'on creation' > specialization on initialize-instance without having it called > multiple times in unexpected ways. Hi all, I've been giving this some thought and without investigating how much work this would take I see the loading of persistent instances working something like this. We (actually in this case i only speak for myself) want something which, a) Allow customization of both instance creation and instance deserialization. b) fits in well with the current initialization & reinitialization protocols. c) Is relatively (for some value of relatively) non obtrusive. With that in mind I would propose something like the following. Deserializing instances is done using recreate-instance. This method allocates a new instance of the class using allocate-instance and passes this to reconstitute-instance which falls through to shared-initialize which does all the dirty work. (recreate-instance (persistent-metaclass)) -> (allocate-instance (standard-class) &rest &key) -> (reconstitute-instance (persistent-object) &rest &key) -> (shared-initialize (persistent-object) t &rest &key) This should satisfy everyones requirements, allowing specialization of methods for instance deserilazation, instance creation or both using initialize-instance, reconstitute-instance and shared-initialize respectively. This would also make it relatively easy to add in schema evolution by adding a :before method on reconstitute instance. The only drawback that is immediately apparent to me is that this can break backwards compatibility since there may be specialized methods on initialize-instance that will no longer be called, although this can be easily worked around by specializing on shared-initialize or altering reconstitute-instance to call initialize-instance. comments? cheers, sean. From killerstorm at newmail.ru Tue Jan 8 19:31:29 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 8 Jan 2008 21:31:29 +0200 Subject: [elephant-devel] Re: recreate-instance-using-class References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu><4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> Message-ID: SR> I'm sorry to hear that my patch caused so many issues with postmodern SR> and I'm quite keen to investigate the cause of this. I'll be SR> installing setting up the postmodern backend and seeing if I can track SR> down the causes of these problems. looks like it's related to some deep weirdness of db-postmodern implementation, and i think i've almost tracked this down, so i don't recommend anyone spending time on this. (well, until/if i'll give up..) however i'd like to see comments about class "persistent" -- it doesn't even recieve shared-initialize, are there any reasons for special threating of it? and are there reasons for using this recreate initialization sequence for class "persistent"? only instances of classes of persistent-metaclass metaclass (e.g. persistent-object) can have persistent slots, so it makes sense to bypass make-instance only for such classes. other classes have quite SR> As to where to go from here I do agree with robert on this and that it SR> is nice to have the seperation between the creation and restoration of SR> instances although I think this needs to be done with a well thought SR> out protocol. SR> cheers, SR> sean. ) (With-best-regards '(Alex Mizrahi) :aka 'killer_storm) "Hanging In The Balance Of Deceit And Blasphemy") From killerstorm at newmail.ru Tue Jan 8 19:33:53 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 8 Jan 2008 21:33:53 +0200 Subject: [elephant-devel] Re: recreate-instance-using-class References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu><4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> Message-ID: SR> I'm sorry to hear that my patch caused so many issues with postmodern SR> and I'm quite keen to investigate the cause of this. I'll be SR> installing setting up the postmodern backend and seeing if I can track SR> down the causes of these problems. looks like it's related to some deep weirdness of db-postmodern implementation, and i think i've almost tracked this down, so i don't recommend anyone spending time on this. (well, until/if i'll give up..) however i'd like to see comments about class "persistent" -- it doesn't even recieve shared-initialize, are there any reasons for special threating of it? and are there reasons for using this recreate initialization sequence for class "persistent"? only instances of classes of persistent-metaclass metaclass (e.g. persistent-object) can have persistent slots, so it makes sense to bypass make-instance only for such classes. so i'm thinking about patching this recreation stuff this way: ;; ;; RECREATING A PERSISTENT INSTANCE ;; (defmethod recreate-instance-using-class ((class standard-class) &rest initargs &key &allow-other-keys) "recreate-instance-using-class uses normal initialization sequence for ordinary classes." (apply #'make-instance class initargs)) (defmethod recreate-instance-using-class ((class persistent-metaclass) &rest initargs &key &allow-other-keys) "objects having persistent metaclass bypass normal initialization sequence when they get deserialized. go figure how to do 'transient' initialization for them.." (let ((instance (allocate-instance class))) (apply #'recreate-instance instance initargs) instance)) (defgeneric recreate-instance (instance &rest initargs &key &allow-other-keys) (:method ((instance persistent-object) &rest args &key from-oid (sc *store-controller*)) (initial-persistent-setup instance :from-oid from-oid :sc sc) (shared-initialize instance t :from-oid from-oid))) what do you think of it? From read at robertlread.net Tue Jan 8 19:38:20 2008 From: read at robertlread.net (Robert L. Read) Date: Tue, 08 Jan 2008 13:38:20 -0600 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> <4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> Message-ID: <1199821100.3783.26.camel@penguin.yourdomain.com> Thank you very much, Sean. I suggest you work with Henrik and Alex and I on this as well (possibly off-line, outside of this forum.) In case it helps, here is what I tried to do, and what I think I discovered. Please forgive me if I am inconsistent; I clearly have been confused on some of these points. This has been a learning process for me; I am writing this only in hopes of saving you time. 1) It so happens that some of the elephant objects do exactly what we are discussing --- that is, they do special initialization in "shared-initialize". 2) The test always worked for me (the first time they were run.) When I ran them the second time, I had failures. I am not 100% sure this had to do with your patch or not...it is possible that I have an SBCL error or some other problem. 3) It seems straightforward to rewrite the "make-instance" and "shared-initialize" methods to separate things into an auxiliary initialization function called by "make-instance" and "recreate-instance-using-class". However, this is tricky when it comes to pm-indexed-btree, the classes that implement the secondary cursors. There may be some danger that "persistent-slot-writer" puts us in a foward-reference problem: You're trying to initialize the class but the initialization code assumes that make-instance has already been called, but something it requires is not initialized. I think if I clear headed-person understood exactly the object creation protocol, they could probably refactor it out successfully. 4) Given the multiple inheritance of "pm-indexed-btree" (a subclass of "indexed-btree" and "pm-btree", it was not clear to me exactly what order the MOP stuff got called. It is possible that some subtle aspects of this are even lisp-dependent. So, in summary, what I tried to do was to rewrite the files in src/db-postmodern so that they were properly initialized by recreate-instance-using-class. I was never able to get this to work, and whether that is due to my own confusion (I have bronchitis at the moment) or some weirdness in SBCL I can't say. I suggest you and Ian come up with the "reconstitution" protocol which he has proposed. I'm happy to work to on this, but I don't feel that I understand common usage of LISP --- all of my coding has in LISP has been in complete isolation outside of this list. On Tue, 2008-01-08 at 17:45 +0000, Sean Ross wrote: > Hi all, > > I'm sorry to hear that my patch caused so many issues with postmodern > and I'm quite keen to investigate the cause of this. I'll be > installing setting up the postmodern backend and seeing if I can track > down the causes of these problems. > > As to where to go from here I do agree with robert on this and that it > is nice to have the seperation between the creation and restoration of > instances although I think this needs to be done with a well thought > out protocol. > > cheers, > sean. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Tue Jan 8 20:12:49 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 8 Jan 2008 15:12:49 -0500 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu><4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> Message-ID: <51DC9C57-B2A2-40D3-A585-C2FE549DD200@csail.mit.edu> persistent is simply a base class that maintains an OID which is initialized using the :from-oid (so it picks up the oid during deserialization by default) persistent-object is the superclass for all objects for which we are allowing the persistent slot access protocol persistent-collection is essentially a placeholder. Persistent collection objects do not have persistent slots, but act as types to specialize gf's like get-value. persistent-collection inherits oid and the :from-oid initarg from persistent. Cool! I apologize that I really don't have time to fully think through your code just now so I might get some of this wrong as it's mostly off the top of my head, however it seems like you've got the right idea. 1) The call to shared-initialize (and the :around method on persistent- object) could cause some of the problems people have reported. For example, if I make a slot unbound on an instance, when it is reloaded, shared-initialize will evaluate the default form and rebind it to that value. If the initform is at all interesting, rather than a default value, then that function can get called more than once leading to potentially unexpected side-effects. FIX?: If you call shared-initialize with the slotnames returned by (transient-slot-names class) this might avoid those problems and handle the transient slot init all in one go. 2) This code seems to bypass initialize-instance :before which sets up the connection to the home store controller and I think is responsible for setting the oid. but then we've called the :around function on persistent-objects which calls instance initialization for all the slots. FIX: We should factor out this functionality into a helper function called 'associate-persistent-object' (or perhaps this what you intended by initial-persistence-setup? You didn't provide a definition for it) and put a call to the helper function in initialize- instance :before as well as after the allocate-instance call in recreate-instance-using-class The functions for persistent-objects are: - initialize-instance :before sets up the OIDs and store-controller link for the instance called in both cases has to be called prior to shared-initialize :around in the creation case - shared-initialize :around initializes both persistent and transient slot values should be called only during creation PROBLEM: I think the standard shared-initialize needs to be called during deserialization too? - update-instance-for-redefined-class called by CLOS; should be OK, but isn't applied properly to all db instances so needs some schema-evolution support - change-class / update-instance-for-different-class FIX! This should be specialized on persistent-object, not persistent I think Does it make any sense today to change the class of a btree? Perhaps we should keep it as it is and catch the cases where a user tries this on a persistent-collection - slot protocol relies on the oid/controller being set, but should be good until we get into slot value caching I guess this wasn't as hard as I thought! Ian On Jan 8, 2008, at 2:31 PM, Alex Mizrahi wrote: > SR> I'm sorry to hear that my patch caused so many issues with > postmodern > SR> and I'm quite keen to investigate the cause of this. I'll be > SR> installing setting up the postmodern backend and seeing if I can > track > SR> down the causes of these problems. > > looks like it's related to some deep weirdness of db-postmodern > implementation, and i think i've almost tracked this down, so i don't > recommend anyone spending time on this. (well, until/if i'll give > up..) > > however i'd like to see comments about class "persistent" -- it > doesn't even > recieve shared-initialize, are there any reasons for special > threating of > it? > > and are there reasons for using this recreate initialization > sequence for > class "persistent"? > > only instances of classes of persistent-metaclass metaclass (e.g. > persistent-object) can have persistent slots, so it makes sense to > bypass > make-instance only for such classes. > other classes have quite > > SR> As to where to go from here I do agree with robert on this and > that it > SR> is nice to have the seperation between the creation and > restoration of > SR> instances although I think this needs to be done with a well > thought > SR> out protocol. > > SR> cheers, > SR> sean. > > ) > (With-best-regards '(Alex Mizrahi) :aka 'killer_storm) > "Hanging In The Balance Of Deceit And Blasphemy") > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From rosssd at gmail.com Tue Jan 8 20:35:14 2008 From: rosssd at gmail.com (Sean Ross) Date: Tue, 8 Jan 2008 20:35:14 +0000 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: <1199821100.3783.26.camel@penguin.yourdomain.com> References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> <4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> <1199821100.3783.26.camel@penguin.yourdomain.com> Message-ID: <5bef28df0801081235u1d1dfb6br5350e9205e470611@mail.gmail.com> On Jan 8, 2008 7:38 PM, Robert L. Read wrote: > Thank you very much, Sean. I suggest you work with Henrik and Alex and > I on this as well (possibly off-line, outside of this forum.) Thanks, I'll try and put together an initial proposal for Henrik, Alex and I to start with. > 2) The test always worked for me (the first time they were run.) When > I ran them the second time, I had failures. I am not 100% sure this had > to do with your patch or not...it is possible that I have an SBCL error > or some other problem. I'm hoping that when I get postmodern and the tests up and running that I can test this with Lispworks. > 4) Given the multiple inheritance of "pm-indexed-btree" (a subclass of > "indexed-btree" and "pm-btree", it was not clear to me exactly what > order the MOP stuff got called. It is possible that some subtle aspects > of this are even lisp-dependent. I think I may have discovered why pm-indexed-btree could behave oddly with the new patch. All the methods defined are defined on persistent-object and it appears that persistent-object is not on pm-indexed-btree's class-precedence-list which could cause various amounts of havoc. Alex, can you confirm/refute this? On a different note, It may be worth requiring that perstistent-metaclasses have persistent-object on their class-precedence-list although that is bound to have implications that I haven't thought of. sean. From rosssd at gmail.com Tue Jan 8 21:03:26 2008 From: rosssd at gmail.com (Sean Ross) Date: Tue, 8 Jan 2008 21:03:26 +0000 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: <5bef28df0801081235u1d1dfb6br5350e9205e470611@mail.gmail.com> References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> <4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> <1199821100.3783.26.camel@penguin.yourdomain.com> <5bef28df0801081235u1d1dfb6br5350e9205e470611@mail.gmail.com> Message-ID: <5bef28df0801081303t6db1e984j8e271f2309356aff@mail.gmail.com> On Jan 8, 2008 8:35 PM, Sean Ross wrote: > On a different note, It may be worth requiring that > perstistent-metaclasses have persistent-object on their class-precedence-list > although that is bound to have implications that I haven't thought of. Sorry all, please ignore that last brainfart, this is already in place. sean. From killerstorm at newmail.ru Tue Jan 8 21:08:19 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 8 Jan 2008 23:08:19 +0200 Subject: [elephant-devel] Re: recreate-instance-using-class References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu><4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu><5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com><1199821100.3783.26.camel@penguin.yourdomain.com> <5bef28df0801081235u1d1dfb6br5350e9205e470611@mail.gmail.com> Message-ID: SR> All the methods defined are defined on persistent-object and it SR> appears that persistent-object SR> is not on pm-indexed-btree's class-precedence-list which could cause SR> various amounts of havoc. doesn't this function do so: (defmethod shared-initialize :around ((class persistent-metaclass) slot-names &rest args &key direct-superclasses index) "Ensures we inherit from persistent-object." ? From read at robertlread.net Tue Jan 8 21:00:54 2008 From: read at robertlread.net (Robert L. Read) Date: Tue, 08 Jan 2008 15:00:54 -0600 Subject: [elephant-devel] bugfix for db-bdb: utf16 string comparison In-Reply-To: <87ir2gdy9w.fsf@kzn.homelinux.org> References: <87ir2gdy9w.fsf@kzn.homelinux.org> Message-ID: <1199826054.3783.30.camel@penguin.yourdomain.com> Thank you, Anton. I'm sorry it has taken so long to get to this (the holidays clog things up.) I applied this patch and was green under SBCL on postmodern and postgres. I can't test under BDB, but I assumed this was enough evidence to commit it to the repository, so I have done so. If anybody notices a problem, please let me know --- this will be tested again when we do the release, but it always better to keep the branch as clean as possible. On Sun, 2007-12-30 at 17:59 +0300, Anton Kazennikov wrote: > Hello all! > > I found a bug in db-bdb string comparison functions - when comparing > two utf16 strings only the first half of strings are compared. (bug is in > misinterpreting string size in chars and in bytes). > plain text document attachment (bdb-bugfix.txt), "darcs-patch.txt" > New patches: > > [db-bdb bugfix: when bdb key comparison compared only the first half of utf16 strings > kazennikov at gmail.com**20071230141055] { > hunk ./src/db-bdb/libberkeley-db.c 1257 > - limit1=s1+length1; > + limit1=s1+2*length1; > hunk ./src/db-bdb/libberkeley-db.c 1260 > - limit1=s1+length1; > + limit1=s1+2*length1; > hunk ./src/db-bdb/libberkeley-db.c 1263 > - limit1=s1+length2; > + limit1=s1+2*length2; > } > > Context: > > [db-postmodern: optimized map-index for -by-value case > alex.mizrahi at gmail.com**20071207195402] > [db-postmodern: optimized form-slot-key for persistent-slot-reader > alex.mizrahi at gmail.com**20071207200835 > it uses SBCL internal function now, for other implementation it's less optimized. > ] > [db-postmodern: small example update > alex.mizrahi at gmail.com**20071207200630] > [added sh script for flushing logs sample > alex.mizrahi at gmail.com**20070920095806] > [db-postmodern removed possiblity of using NIL as a key in btrees > Henrik Hjelte**20071124163828] > [cursor-duplicate removed from db-postmodern > Henrik Hjelte**20071124163701] > [removed a little compiler warning (typo) > Henrik Hjelte**20071122151929] > [remove kind-hints parameter from add-index > Henrik Hjelte**20071122151046 > Probably a coming feature from Ian, but > right now it breaks the generic function add-index > and thus postmodern, so I removed it for now. > ] > [Ensure set-db-synch is defined before pset is loaded > sross at common-lisp.net**20071214145041] > [Fix instance deserialization to bypass initialization protocol > sross at common-lisp.net**20071214141938] > [Fix to from-end traversal of new map-index > eslick at common-lisp.net**20071130223524] > [New map-index implementation > eslick at common-lisp.net**20071130222620] > [Cheaper get-instance-by-value > eslick at common-lisp.net**20071130222520] > [TAG ELEPHANT-0-9-1 > ieslick at common-lisp.net**20071116153634] > Patch bundle hash: > c4410bd1cd704cb2e4eb5eede85d1d4961621945 > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Tue Jan 8 21:52:12 2008 From: read at robertlread.net (Robert L. Read) Date: Tue, 08 Jan 2008 15:52:12 -0600 Subject: [elephant-devel] Fix the use of internal symbol of sb-kernel in memutils In-Reply-To: <5d9c2e640712291624t5e26ffc0m45fd4bc249f764cf@mail.gmail.com> References: <5d9c2e640712291624t5e26ffc0m45fd4bc249f764cf@mail.gmail.com> Message-ID: <1199829132.3783.42.camel@penguin.yourdomain.com> Thank you. I have committed this patch to the repository. However, under sbcl on a x86_64 architecture, I have to build libmemutil by hand, because of the errors below. I have no way to explain these. After executing gcc by hand, it is not a problem. This is the REPL error: Implicitly creating new generic function ENQUEUE-OR-MOVE-TO-FRONT. ; $ gcc -shared -march=x86-64 -fPIC -Wall -O2 -g /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.c -o /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.so -lm And this is the backtrace is below: Unhandled memory fault at #x2AAAB1CB2000. [Condition of type SB-SYS:MEMORY-FAULT-ERROR] Restarts: 0: [RETRY] Retry performing # on #. 1: [ACCEPT] Continue, treating # on # as having been successful. 2: [ABORT] Return to SLIME's top level. 3: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-SYS:MEMORY-FAULT-ERROR) 1: (SB-SYS:MEMORY-FAULT-ERROR) 2: ("foreign function: #x41C312") 3: ("foreign function: #x41C3E0") 4: (SB-KERNEL:COPY-UB8-TO-SYSTEM-AREA :INVALID-VALUE-FOR-UNESCAPED-REGISTER-STORAGE 0 #.(SB-SYS:INT-SAP #X2AAAB1CB1FF0) 0 19) 5: (SB-IMPL::STRING-LIST-TO-C-STRVEC ("PWD=/home/read/projects/kbench/sql-back-end/elephant/tests" "TERM=dumb" "TERMCAP=" "COLUMNS=95" "EMACS=t" "COLORTERM=gnome-terminal" "XAUTHORITY=/home/read/.Xauthority" "G_BROKEN_FILENAMES=1" "DISPLAY=:0.0" "LESSOPEN=|/usr/bin/lesspipe.sh %s" ...)) 6: ((LAMBDA ())) 7: ((FLET SB-THREAD::%CALL-WITH-SYSTEM-MUTEX)) 8: (SB-UNIX::CALL-ALLOWING-WITH-INTERRUPTS # T) 9: ((FLET SB-UNIX::WITHOUT-INTERRUPTS-THUNK) T) 10: ((FLET SB-UNIX::RUN-WITHOUT-INTERRUPTS)) 11: (SB-UNIX::CALL-WITHOUT-INTERRUPTS #) 12: (SB-THREAD::CALL-WITH-SYSTEM-MUTEX # #S(SB-THREAD:MUTEX :NAME "Lock for active processes." :%OWNER # :STATE 1) NIL) 13: (RUN-PROGRAM "/bin/sh" ("-c" "gcc -shared -march=x86-64 -fPIC -Wall -O2 -g /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.c -o /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.so -lm ")) 14: (ASDF:RUN-SHELL-COMMAND "~A ~{~A ~}") 15: ((SB-PCL::FAST-METHOD ASDF:PERFORM (ASDF:COMPILE-OP ELEPHANT-SYSTEM:ELEPHANT-C-SOURCE)) # # # #) 16: ((LAMBDA (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. SB-PCL::.ARG1.)) # # # #) 17: ((LAMBDA ())) 18: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK)) 19: (SB-UNIX::CALL-WITH-LOCAL-INTERRUPTS # T) --more-- On Sat, 2007-12-29 at 22:24 -0200, Leonardo Varuzza wrote: > Hello, > > I found a minor problem compiling memutil.lisp , because the > functions sb-kernel::copy-*-from-system-area aren't exported in the > latest version of sbcl. > > I made a little patch to fix it. > > Best Regards, > Leonardo Varuzza. > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From Alain.Picard at memetrics.com Tue Jan 8 22:13:42 2008 From: Alain.Picard at memetrics.com (Alain Picard) Date: Wed, 09 Jan 2008 09:13:42 +1100 Subject: [elephant-devel] recreate-instance-using-class In-Reply-To: <1199808929.3783.1.camel@penguin.yourdomain.com> (Robert L. Read's message of "Tue, 08 Jan 2008 10:15:29 -0600") References: <1199803295.3740.209.camel@penguin.yourdomain.com> <75FD8FA7-E4B7-42E6-9E5C-CD3838E1387B@csail.mit.edu> <1199808929.3783.1.camel@penguin.yourdomain.com> Message-ID: <87tzlo9d9l.fsf@memetrics.com> "Robert L. Read" writes: > The contract is that make-instance should only be called once for a > given object? FWIW, I wrote a CLOS->SQL layer a few years ago, and I do agreed with this assessment (that MAKE-INSTANCE should only get called once), so my protocol ended up with these two extra methods: (defgeneric customize-persistent-instance (instance &rest initargs &key &allow-other-keys) (:documentation "Used to customize an instance when it is created. This takes the place of the normal initialize-instance :after method. This will be called both when the objects is being created for the first time and being re-selected. Subclasses should define :after methods as required.") (:method (instance &rest initargs &key &allow-other-keys) t)) (defgeneric customize-new-persistent-instance (instance &rest initargs &key &allow-other-keys) (:documentation "Used to customize an instance when it is newly created by the application. The object HAS been commited to the database.") (:method (instance &rest initargs &key &allow-other-keys) t)) In my mind, when you get an instance via a SELECT, you're not calling MAKE-INSTANCE, you're doing a hash table lookup to find an object stored persistently in some imaginary infinitely large lisp image. YMMV, of course. --ap From varuzza at gmail.com Wed Jan 9 02:56:19 2008 From: varuzza at gmail.com (Leonardo Varuzza) Date: Wed, 9 Jan 2008 00:56:19 -0200 Subject: [elephant-devel] Fix the use of internal symbol of sb-kernel in memutils In-Reply-To: <1199829132.3783.42.camel@penguin.yourdomain.com> References: <5d9c2e640712291624t5e26ffc0m45fd4bc249f764cf@mail.gmail.com> <1199829132.3783.42.camel@penguin.yourdomain.com> Message-ID: <5d9c2e640801081856w621f755bmbae1497c8360d3fb@mail.gmail.com> After your e-mail I installed elephant in the my Atlhon64 linux box, using the latest CVS version of SBCL. The memutil compiled automaticaly, but I get a diferent error when trying to open-store: Unhandled memory fault at #x4020002. [Condition of type SB-SYS:MEMORY-FAULT-ERROR] Restarts: 0: [ABORT-REQUEST] Abort handling SLIME request. 1: [TERMINATE-THREAD] Terminate this thread (#) Backtrace: 0: (SB-SYS:MEMORY-FAULT-ERROR) 1: (SB-SYS:MEMORY-FAULT-ERROR) 2: ("foreign function: #x41D8F2") 3: ("foreign function: #x41D9C0") 4: ("foreign function: #x2B42E392B236") 5: (DB-BDB::DB-ENV-CREATE) 6: ((SB-PCL::FAST-METHOD ELEPHANT::OPEN-CONTROLLER (DB-BDB::BDB-STORE-CONTROLLER)) #) 7: (OPEN-STORE (:BDB "/home/lucien/lisp/store/")) 8: (SB-INT:SIMPLE-EVAL-IN-LEXENV (OPEN-STORE (QUOTE (:BDB "/home/lucien/lisp/store/"))) #) 9: (SWANK::EVAL-REGION "(open-store '(:BDB \"/home/lucien/lisp/store/\")) In the Intel macbook everything works fine. On Jan 8, 2008 7:52 PM, Robert L. Read wrote: > Thank you. I have committed this patch to the repository. > > > > > However, under sbcl on a x86_64 architecture, I have to build libmemutil > by hand, because of the errors below. I have no way to explain these. > > After executing gcc by hand, it is not a problem. > > > This is the REPL error: > Implicitly creating new generic function ENQUEUE-OR-MOVE-TO-FRONT. > ; $ gcc -shared -march=x86-64 -fPIC -Wall -O2 > -g /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.c -o /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.so -lm > > And this is the backtrace is below: > > > Unhandled memory fault at #x2AAAB1CB2000. > [Condition of type SB-SYS:MEMORY-FAULT-ERROR] > > Restarts: > 0: [RETRY] Retry performing # on > #. > 1: [ACCEPT] Continue, treating # on > # as having > been successful. > 2: [ABORT] Return to SLIME's top level. > 3: [TERMINATE-THREAD] Terminate this thread (# "repl-thread" {10030C1DA1}>) > > Backtrace: > 0: (SB-SYS:MEMORY-FAULT-ERROR) > 1: (SB-SYS:MEMORY-FAULT-ERROR) > 2: ("foreign function: #x41C312") > 3: ("foreign function: #x41C3E0") > 4: (SB-KERNEL:COPY-UB8-TO-SYSTEM-AREA > :INVALID-VALUE-FOR-UNESCAPED-REGISTER-STORAGE > 0 > #.(SB-SYS:INT-SAP #X2AAAB1CB1FF0) > 0 > 19) > 5: (SB-IMPL::STRING-LIST-TO-C-STRVEC > ("PWD=/home/read/projects/kbench/sql-back-end/elephant/tests" > "TERM=dumb" "TERMCAP=" "COLUMNS=95" "EMACS=t" > "COLORTERM=gnome-terminal" "XAUTHORITY=/home/read/.Xauthority" > "G_BROKEN_FILENAMES=1" "DISPLAY=:0.0" > "LESSOPEN=|/usr/bin/lesspipe.sh %s" ...)) > 6: ((LAMBDA ())) > 7: ((FLET SB-THREAD::%CALL-WITH-SYSTEM-MUTEX)) > 8: (SB-UNIX::CALL-ALLOWING-WITH-INTERRUPTS > # {2AAAB1A63549}> > T) > 9: ((FLET SB-UNIX::WITHOUT-INTERRUPTS-THUNK) T) > 10: ((FLET SB-UNIX::RUN-WITHOUT-INTERRUPTS)) > 11: (SB-UNIX::CALL-WITHOUT-INTERRUPTS > # {2AAAB1A63729}>) > 12: (SB-THREAD::CALL-WITH-SYSTEM-MUTEX > # > #S(SB-THREAD:MUTEX > :NAME "Lock for active processes." > :%OWNER # > :STATE 1) > NIL) > 13: (RUN-PROGRAM > "/bin/sh" > ("-c" > "gcc -shared -march=x86-64 -fPIC -Wall -O2 > -g /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.c -o /home/read/projects/kbench/sql-back-end/elephant/src/memutil/libmemutil.so -lm ")) > 14: (ASDF:RUN-SHELL-COMMAND "~A ~{~A ~}") > 15: ((SB-PCL::FAST-METHOD ASDF:PERFORM > (ASDF:COMPILE-OP ELEPHANT-SYSTEM:ELEPHANT-C-SOURCE)) > # > # > # > #) > 16: ((LAMBDA > (SB-PCL::.PV. SB-PCL::.NEXT-METHOD-CALL. SB-PCL::.ARG0. > SB-PCL::.ARG1.)) > # > # > # > #) > 17: ((LAMBDA ())) > 18: ((FLET SB-THREAD::WITH-RECURSIVE-LOCK-THUNK)) > 19: (SB-UNIX::CALL-WITH-LOCAL-INTERRUPTS > # {2AAAB1A63D39}> > T) > --more-- > > > On Sat, 2007-12-29 at 22:24 -0200, Leonardo Varuzza wrote: > > Hello, > > > > I found a minor problem compiling memutil.lisp , because the > > functions sb-kernel::copy-*-from-system-area aren't exported in the > > latest version of sbcl. > > > > I made a little patch to fix it. > > > > Best Regards, > > Leonardo Varuzza. > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -- The most fundamental particles in this product are held together by a "gluing" force about which little is currently known and whose adhesive power can therefore not be permanently guaranteed. From killerstorm at newmail.ru Wed Jan 9 09:02:06 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 9 Jan 2008 11:02:06 +0200 Subject: [elephant-devel] fix of ele-postmodern to work with recreate-instance-using-class Message-ID: the problem was actually quite easy -- pm-btree needs to initialize it's transient each time it's created or deserialized. it participates both as descendant from "persistent" as pm-btree itself, and as persistent-object in form of pm-indexed-btree. (so pm-btree is quite a good test for a new system) as we've found out, now initialization of transient slots should be done in shared-initialize :after (it's funny, nobody have replied directly to my concrete question, but it was mentioned in some 'proposals'). unfortunately original implementation of recreate-instance-using-class deprives "persistent" from any kind of initialization, including shared-initialize. i believe it's a bug and recreate-instance-using-class either should call normal make-instance for everything except persistent-object. or, at least, it should call shared-initialize. (again, this was in my original question but nobody addressed this issues directly. is my English that poor so nobody understand me??). so, fix consists of two parts: 1. `persistent' gets normal initialization via make-instance (i've already posted code), so pm-btree has chance to initialize itself; 2. initialize-instance :after of pm-btree is changed to shared-initialize so it will work for pm-indexed-btree which is persistent-object. does this make sense according to original design? if it does, i'll send a patch. if part 1 is against the spirit of this recreate thing we can just add shared-initialize call in recreate-instance of `persistent'. i don't see how it will change anything though.. p.s. fixing this was somewhat complicated because it didn't clearly report problem that pm-btree is not initialized, but tried to initialize itself in erroneous and confusing way in some obsoleted code path. i'm going to delete these code pathes, so we won't be confused if something like that happens again From read at robertlread.net Wed Jan 9 14:42:54 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 09 Jan 2008 08:42:54 -0600 Subject: [elephant-devel] fix of ele-postmodern to work with recreate-instance-using-class In-Reply-To: References: Message-ID: <1199889774.3783.54.camel@penguin.yourdomain.com> On Wed, 2008-01-09 at 11:02 +0200, Alex Mizrahi wrote: > is my English that poor so nobody understand me??). > No, your English is great. I think there were so many emails yesterday that it was hard to know exactly whom to respond to. Thanks, this is great work; I'm sorry I couldn't figure it out. I'll let Sean and Ian answer your question about whether your proposed solution is in the spirit of their ideas or not. From rosssd at gmail.com Wed Jan 9 15:15:13 2008 From: rosssd at gmail.com (Sean Ross) Date: Wed, 9 Jan 2008 15:15:13 +0000 Subject: [elephant-devel] Re: recreate-instance-using-class In-Reply-To: References: <40C2C689-3FB8-426A-8F82-228FEA0BA080@csail.mit.edu> <4CFAA63A-9BAC-4591-9775-5640786D425C@csail.mit.edu> <5bef28df0801080945m4e8855a2mdd9f4091148a961f@mail.gmail.com> Message-ID: <5bef28df0801090715x4c5e0aa7wb56ecd93ac9c5b3@mail.gmail.com> On Jan 8, 2008 7:33 PM, Alex Mizrahi wrote: > so i'm thinking about patching this recreation stuff this way: > > ;; > ;; RECREATING A PERSISTENT INSTANCE > ;; > > (defmethod recreate-instance-using-class ((class standard-class) &rest > initargs &key &allow-other-keys) > "recreate-instance-using-class uses normal initialization sequence for > ordinary classes." > (apply #'make-instance class initargs)) > > (defmethod recreate-instance-using-class ((class persistent-metaclass) &rest > initargs &key &allow-other-keys) > "objects having persistent metaclass bypass normal initialization sequence > when they get deserialized. > go figure how to do 'transient' initialization for them.." > (let ((instance (allocate-instance class))) > (apply #'recreate-instance instance initargs) > instance)) > > (defgeneric recreate-instance (instance &rest initargs &key > &allow-other-keys) > (:method ((instance persistent-object) &rest args &key from-oid (sc > *store-controller*)) > (initial-persistent-setup instance :from-oid from-oid :sc sc) > (shared-initialize instance t :from-oid from-oid))) > > what do you think of it? sorry for the delay, Looks good to me although I'm now of the opinion that (recreate-instance class) and (reconstitute-instance object) are better names as they fit in with the current naming and argument conventions of CLOS. sean. From eslick at csail.mit.edu Wed Jan 9 15:26:32 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 9 Jan 2008 10:26:32 -0500 Subject: [elephant-devel] fix of ele-postmodern to work with recreate-instance-using-class In-Reply-To: References: Message-ID: <52CE1FAD-007D-45A1-B55E-5D8FE94974E1@csail.mit.edu> On Jan 9, 2008, at 4:02 AM, Alex Mizrahi wrote: > the problem was actually quite easy -- pm-btree needs to initialize > it's > transient each time it's created or deserialized. it participates > both as > descendant from "persistent" as pm-btree itself, and as persistent- > object in > form of pm-indexed-btree. I think the original idea is that persistent-collections are special and not supposed to be persistent-objects, but this conflicts with the default behavior of the persistent-metaclass 'ensure persistent- object' functionality. This may have been some of the problem. > (so pm-btree is quite a good test for a new system) > > as we've found out, now initialization of transient slots should be > done in > shared-initialize :after (it's funny, nobody have replied directly > to my > concrete question, but it was mentioned in some 'proposals'). This is how it was done for bdb-indexed-btree > unfortunately original implementation of recreate-instance-using-class > deprives "persistent" from any kind of initialization, including > shared-initialize. i believe it's a bug and recreate-instance-using- > class > either should call normal make-instance for everything except > persistent-object. or, at least, it should call shared-initialize. make-instance for standard classes is probably just fine; although this may have some of the same problems - that an object init function is called on each deserialization. Still, this is normal since standard classes really are 'recreated' when retrieved from the store as they haven't been made 'persistent' and so users should design their systems accordingly. As you note below, this handles the missing case of initializing persistent for non persistent-metaclass objects that are persistent (i.e. persistent-collections) > > (again, this was in my original question but nobody addressed this > issues > directly. is my English that poor so nobody understand me??). I think part of the problem is that none of us have our heads around all the details you've been working on so intimately. I've found that usage of the MOP leads to the need to model alot of hidden state. It takes me awhile to remember all the details since it's been almost a year since I worked on this part of Elephant in earnest. In general, you seem to be uncovering some real problems with the semantics and use of persistent, persistent-object and persistent- collection and their interaction with persistent-metaclass. btrees are standard-classes that also have 'persistent' on them so they get serialized properly as OID + class, but don't have the slot protocol defined on them so aren't persistent-objects or part of the persistent-metaclass (they aren't indexed, etc). Since indexed-btrees need to have an additional persistent parameter, they were made a persistent-metaclass which means they inherit from both persistent-collections and persistent-object. persistent-objects should get initialized via (shared-initialize instance (transient-slot-names (class-of instance)) ...) to initialize only the transient slots (this will catch pm-indexed-btree slots but not cause initforms to be re-evaluated for any unbound persistent slots. I think this is inline with the general idea, but it would be nice to hear what the correct semantics of the subclasses of persistent and their interaction with persistent-metaclass should be! You're probably the most qualified to answer this now. I don't have a strong opinion about how this should be fixed, but let's make sure we aren't hacking up the initialization to fix a fundamental problem with the use of the class hierarchy. It might make more sense if we make indexed-btrees not inherit from persistent objects...but that would mean special functionality is needed from every data store to maintain persistent properties and values attached to persistent-collection objects... If the distinction isn't useful, we should just collapse everything to 'persistent'. > so, fix consists of two parts: > 1. `persistent' gets normal initialization via make-instance (i've > already > posted code), so pm-btree has chance to initialize itself; This sounds right. > 2. initialize-instance :after of pm-btree is changed to shared- > initialize > so it will work for pm-indexed-btree which is persistent-object. This is good. > does this make sense according to original design? if it does, i'll > send a > patch. > > if part 1 is against the spirit of this recreate thing we can just add > shared-initialize call in recreate-instance of `persistent'. > i don't see how it will change anything though.. > > p.s. fixing this was somewhat complicated because it didn't clearly > report > problem that pm-btree is not initialized, but tried to initialize > itself in > erroneous and confusing way in some obsoleted code path. i'm going > to delete > these code pathes, so we won't be confused if something like that > happens > again > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Wed Jan 9 17:57:06 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Wed, 9 Jan 2008 19:57:06 +0200 Subject: [elephant-devel] Re: fix of ele-postmodern to work withrecreate-instance-using-class References: <52CE1FAD-007D-45A1-B55E-5D8FE94974E1@csail.mit.edu> Message-ID: IE> I don't have a strong opinion about how this should be fixed, but IE> let's make sure we aren't hacking up the initialization to fix a IE> fundamental problem with the use of the class hierarchy. It might IE> make more sense if we make indexed-btrees not inherit from persistent IE> objects...but that would mean special functionality is needed from IE> every data store to maintain persistent properties and values attached IE> to persistent-collection objects... as for me current class hierarchy looks just fine. separation on low-level `persistent' and higl-level `persistent-object' seems to be reasonable: this adds some flexibility and allows us to implement stuff in unified way. probably it's possible to make some other "better hierarchy", but i don't think that will actually make something actually better, because for now internal stuff we have works fine, and end-user are supposed to interfact only with `persistent-object', as i understand, so they won't notice internal changes anyway. IE> If the distinction isn't useful, we should just collapse everything to IE> 'persistent'. aside from internal implementation details, i think distinction could be a benefit for some advanced use cases -- some people might want to implement their own kind of persistent objects that is different from implementation of persistent-object. they can get lower level `persistent' and implement this as addon, without changing elephant's internals. now, about fixes.. as i've understand we've got few enhancement suggestions -- Sean suggests to change names of functions, and you say the way shared-initialize is called should be changed. and there were suggestions about schema modifications.. looks like i won't be able to do all these enhancements :) (i think you or Sean would do it better), so for now i'll send only changes to db-postmodern code and my little fix for `persistent' initialization issues (unless somebody will submit better code by the time i'll be ready to package patches) -- so at least it would be passing tests with this patches. and latter it can be improved further.. and, by the way, a little question about shared-initialize :after thing: some people might need to move "transient initialization" stuff from initialize-instance :after into shared-initialize :after. shared-initialize is called "more frequently" than initialize-instance -- on class redefinitions etc, and it has additional list of slots. i suspect in some cases this could lead to unexpected results. can you formulate some guidelines about writing shared-initialize :after method for "transient initialization" so it won't cause confusion, or there are only general guidelines to implement it carefully according to circumstances? fortunately pm-btree's initialization method already head checks, but perhaps there are more complex cases.. i think it would be nice if examples on using initialization methods would be included in documentation of next release From rosssd at gmail.com Thu Jan 10 15:37:12 2008 From: rosssd at gmail.com (Sean Ross) Date: Thu, 10 Jan 2008 15:37:12 +0000 Subject: [elephant-devel] function calls as keyforms for btree-indexes Message-ID: <5bef28df0801100737t20a0edbao11db167e9b691ad6@mail.gmail.com> Hi all, I would like to propose the following adjustment to the way that key-forms for btree indexes are converted into functions. Currently this is done by looking up the fdefinition of the symbol provided or calling compile nil on the form. I would like to amend this to also allow function forms eg. '(create-indexer "foo") Keyform lookup would then work as follows. a) (and (symbolp key) (fboundp key)) => (fdefintiion key) b) (and (consp key) (eql (first key) 'lambda)) => (compile nil key) c) (consp key) => (apply (first key) (rest key)) d) error 'invalid-keyform and is used something like the following. (defun create-indexer (tag) (lambda (idx k v) (declare (ignore idx k)) (values (find tag (tags-of v)) (tags-of v)))) and indexes using this can be created by using (list 'create-indexer name) as the key-form. While the functionality of this can be accomplished using (compile nil ...) it seems a bit lispier to invoke a function which subsequently returns a function (and also offers a performance advantage). I've attached a patch which implements this on the shared-initalize :after method for btree-index. cheers, sean. -------------- next part -------------- A non-text attachment was scrubbed... Name: funcalls.darcs Type: application/octet-stream Size: 2590 bytes Desc: not available URL: From norman at norman-werner.de Fri Jan 11 07:24:17 2008 From: norman at norman-werner.de (Norman Werner) Date: Fri, 11 Jan 2008 08:24:17 +0100 Subject: [elephant-devel] typo in example code of documentation Message-ID: <200801110824.17501.norman@norman-werner.de> Hello, During trying out elephant for the first time I found a small typo in the documentation regarding the example code to use a cl-sql backend (http://common-lisp.net/project/elephant/doc/CL_002dSQL-Example.html#CL_002dSQL-Example). Instead of (defvar *testpg-path* '(:postgreql "localhost.localdomain" "test" "postgres" "")) it should be: (defvar *testpg-path* '(:postgresql "localhost.localdomain" "test" "postgres" "")) Regards Norman -- Norman Werner Im Sommerwind 1a 70563 Stuttgart From read at robertlread.net Fri Jan 11 15:25:32 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 11 Jan 2008 09:25:32 -0600 Subject: [elephant-devel] typo in example code of documentation In-Reply-To: <200801110824.17501.norman@norman-werner.de> References: <200801110824.17501.norman@norman-werner.de> Message-ID: <1200065132.3783.122.camel@penguin.yourdomain.com> Thank you. I have fixed this in the repository. On Fri, 2008-01-11 at 08:24 +0100, Norman Werner wrote: > Hello, > > During trying out elephant for the first time I found a small typo in the > documentation regarding the example code to use a cl-sql backend > (http://common-lisp.net/project/elephant/doc/CL_002dSQL-Example.html#CL_002dSQL-Example). > > Instead of > (defvar *testpg-path* > '(:postgreql "localhost.localdomain" "test" "postgres" "")) > > it should be: > (defvar *testpg-path* > '(:postgresql "localhost.localdomain" "test" "postgres" "")) > > Regards > > Norman From read at robertlread.net Fri Jan 11 15:32:53 2008 From: read at robertlread.net (Robert L. Read) Date: Fri, 11 Jan 2008 09:32:53 -0600 Subject: [elephant-devel] function calls as keyforms for btree-indexes In-Reply-To: <5bef28df0801100737t20a0edbao11db167e9b691ad6@mail.gmail.com> References: <5bef28df0801100737t20a0edbao11db167e9b691ad6@mail.gmail.com> Message-ID: <1200065573.3783.129.camel@penguin.yourdomain.com> Thank you Sean. I don't feel I have enough experience to really comment on this idea, so I will assume that it is a good one. I appreciate you providing this patch. However, it would be even better if this patch were accompanied with tests, or modifications to the existing test suite, which exercise, demonstrate, and test this feature. Especially since this is an extension of the API, it makes sense that it should be tested. If I were to accept the patch without tests, it would be adding an untested feature, and there would be the danger that in the future it might be removed, for example, since doing so would break none of the tests. So, do you think it might be possible for you to write some tests within our existing framework to cover this functionality? On Thu, 2008-01-10 at 15:37 +0000, Sean Ross wrote: > Hi all, > > I would like to propose the following adjustment to the way that > key-forms for btree indexes are > converted into functions. Currently this is done by looking up the > fdefinition of the symbol provided > or calling compile nil on the form. I would like to amend this to > also allow function forms > eg. '(create-indexer "foo") > > Keyform lookup would then work as follows. > > a) (and (symbolp key) (fboundp key)) => (fdefintiion key) > b) (and (consp key) (eql (first key) 'lambda)) => (compile nil key) > c) (consp key) => (apply (first key) (rest key)) > d) error 'invalid-keyform > > and is used something like the following. > > (defun create-indexer (tag) > (lambda (idx k v) > (declare (ignore idx k)) > (values (find tag (tags-of v)) > (tags-of v)))) > > and indexes using this can be created by using (list 'create-indexer > name) as the key-form. > > While the functionality of this can be accomplished using (compile nil > ...) it seems a > bit lispier to invoke a function which subsequently returns a function > (and also offers > a performance advantage). > > I've attached a patch which implements this on the shared-initalize > :after method for btree-index. > > cheers, > sean. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From rosssd at gmail.com Fri Jan 11 16:34:37 2008 From: rosssd at gmail.com (Sean Ross) Date: Fri, 11 Jan 2008 16:34:37 +0000 Subject: [elephant-devel] function calls as keyforms for btree-indexes In-Reply-To: <1200065573.3783.129.camel@penguin.yourdomain.com> References: <5bef28df0801100737t20a0edbao11db167e9b691ad6@mail.gmail.com> <1200065573.3783.129.camel@penguin.yourdomain.com> Message-ID: <5bef28df0801110834i4320e8dajc6459ab3d4e16bce@mail.gmail.com> On Jan 11, 2008 3:32 PM, Robert L. Read wrote: > Thank you Sean. > > I don't feel I have enough experience to really comment on this idea, so > I will assume that it is a good one. > > I appreciate you providing this patch. However, it would be even better > if this patch were accompanied with tests, or modifications to the > existing test suite, which exercise, demonstrate, and test this feature. > So, do you think it might be possible for you to write some tests within > our existing framework to cover this functionality? Of course, that is a large oversight on my part. I'll get tests and updated documentation done this weekend. Sean From rosssd at gmail.com Mon Jan 14 13:50:55 2008 From: rosssd at gmail.com (Sean Ross) Date: Mon, 14 Jan 2008 13:50:55 +0000 Subject: [elephant-devel] function calls as keyforms for btree-indexes In-Reply-To: <5bef28df0801110834i4320e8dajc6459ab3d4e16bce@mail.gmail.com> References: <5bef28df0801100737t20a0edbao11db167e9b691ad6@mail.gmail.com> <1200065573.3783.129.camel@penguin.yourdomain.com> <5bef28df0801110834i4320e8dajc6459ab3d4e16bce@mail.gmail.com> Message-ID: <5bef28df0801140550w103dd6aevebd3fb8b222c48ca@mail.gmail.com> On Jan 11, 2008 4:34 PM, Sean Ross wrote: > > I appreciate you providing this patch. However, it would be even better > > if this patch were accompanied with tests, or modifications to the > > existing test suite, which exercise, demonstrate, and test this feature. > > So, do you think it might be possible for you to write some tests within > > our existing framework to cover this functionality? Hi, As promised please find the updated funcall-key-form patch with updated documentation and test, it's not much but I hope that it is sufficient. In order to get the elephant tests to run (in Lispworks) I was force to alter the accessors of the Message class to not be in the keyword package, I have attached the patch for this as well. cheers, sean. -------------- next part -------------- A non-text attachment was scrubbed... Name: funcall-key-form.darcs Type: application/octet-stream Size: 4540 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: non-keyword.darcs Type: application/octet-stream Size: 2580 bytes Desc: not available URL: From read at robertlread.net Mon Jan 14 20:01:18 2008 From: read at robertlread.net (Robert L. Read) Date: Mon, 14 Jan 2008 14:01:18 -0600 Subject: [elephant-devel] function calls as keyforms for btree-indexes In-Reply-To: <5bef28df0801140550w103dd6aevebd3fb8b222c48ca@mail.gmail.com> References: <5bef28df0801100737t20a0edbao11db167e9b691ad6@mail.gmail.com> <1200065573.3783.129.camel@penguin.yourdomain.com> <5bef28df0801110834i4320e8dajc6459ab3d4e16bce@mail.gmail.com> <5bef28df0801140550w103dd6aevebd3fb8b222c48ca@mail.gmail.com> Message-ID: <1200340878.3783.284.camel@penguin.yourdomain.com> Thank you. I was green under postmodern and cl-sql on postgres under SBCL, so I have added your patches to the repo. Thanks again. On Mon, 2008-01-14 at 13:50 +0000, Sean Ross wrote: > On Jan 11, 2008 4:34 PM, Sean Ross wrote: > > > I appreciate you providing this patch. However, it would be even better > > > if this patch were accompanied with tests, or modifications to the > > > existing test suite, which exercise, demonstrate, and test this feature. > > > So, do you think it might be possible for you to write some tests within > > > our existing framework to cover this functionality? > > Hi, > > As promised please find the updated funcall-key-form patch with > updated documentation and test, it's not much but I hope that it is > sufficient. > > In order to get the elephant tests to run (in Lispworks) I was force > to alter the > accessors of the Message class to not be in the keyword package, I > have attached > the patch for this as well. > > cheers, > sean. From joubert at joubster.com Wed Jan 16 06:20:14 2008 From: joubert at joubster.com (Joubert Nel) Date: Wed, 16 Jan 2008 01:20:14 -0500 Subject: [elephant-devel] Having trouble getting started with postmodern Message-ID: <1200464414.28583.13.camel@joubert-desktop> Hi, I just got the latest Elephant from darcs as specified at the top of http://common-lisp.net/project/elephant/downloads.html I'm trying to use postmodern and am using the code in /tests as guidance since I can't find any documentation on using postmodern. Unless I'm mistaken, the ele-postmodern.asd file's defsystem appears wrong to me. The part ":depends-on (:postmodern" --> is this right? I get an error when ele-postmodern is loaded by ASDF, complaining about :POSTMODERN. ====== (defsystem ele-postmodern :name "ele-postmodern" :author "Henrik Hjelte " :version "0.6.0" :licence "LLGPL" :description "Elephant postmodern postgresql backend" :components ((:module :src :components ((:module :db-postmodern :components ((:file "package") (:file "pm-sql") (:file "pm-controller") (:file "pm-cache") (:file "pm-transaction") (:file "pm-btree") (:file "pm-cursor") (:file "pm-btree-index") (:file "pm-indexed-btree") (:file "pm-secondary")) :serial t)))) :depends-on (:postmodern :elephant)) ====== Error: component :POSTMODERN not found, required by # [Condition of type ASDF:MISSING-DEPENDENCY] Regards, Joubert From joubert at joubster.com Wed Jan 16 06:28:18 2008 From: joubert at joubster.com (Joubert Nel) Date: Wed, 16 Jan 2008 01:28:18 -0500 Subject: IGNORE Re: [elephant-devel] Having trouble getting started with postmodern In-Reply-To: <1200464414.28583.13.camel@joubert-desktop> References: <1200464414.28583.13.camel@joubert-desktop> Message-ID: <1200464898.28583.16.camel@joubert-desktop> Apologies, my mistake - late night; got POSTMODERN and can now (require :ele-postmodern). Joubert On Wed, 2008-01-16 at 01:20 -0500, Joubert Nel wrote: > Hi, > > I just got the latest Elephant from darcs as specified at the top of > http://common-lisp.net/project/elephant/downloads.html > > I'm trying to use postmodern and am using the code in /tests as guidance > since I can't find any documentation on using postmodern. Unless I'm > mistaken, the ele-postmodern.asd file's defsystem appears wrong to me. > > The part ":depends-on (:postmodern" --> is this right? I get an error > when ele-postmodern is loaded by ASDF, complaining about :POSTMODERN. > > ====== > (defsystem ele-postmodern > :name "ele-postmodern" > :author "Henrik Hjelte " > :version "0.6.0" > :licence "LLGPL" > :description "Elephant postmodern postgresql backend" > > :components > ((:module :src > :components > ((:module :db-postmodern > :components > ((:file "package") > (:file "pm-sql") > (:file "pm-controller") > (:file "pm-cache") > (:file "pm-transaction") > (:file "pm-btree") > (:file "pm-cursor") > (:file "pm-btree-index") > (:file "pm-indexed-btree") > (:file "pm-secondary")) > :serial t)))) > :depends-on (:postmodern > :elephant)) > > ====== > > Error: > component :POSTMODERN not found, required by > # > [Condition of type ASDF:MISSING-DEPENDENCY] > > > Regards, > Joubert > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Fri Jan 18 11:52:13 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Fri, 18 Jan 2008 13:52:13 +0200 Subject: [elephant-devel] instance cache Message-ID: i've noticed one problem with recreate-instance changes -- it have broken instance cache. instances were cached in initialize-instance :before, which was bypassed, and no instances were cached. moving cache-instance into intial-persistent-setup fixed this, but i'm not sure if it's right place.. this problem broke get-instances-by-value when value is a persistent object. postmodern backend (and i think CLSQL too) use lisp-compare-equal/equalp to check if objects are same in cursor-set, and without instance cache persistent objects were deserialialized into different objects, so they are not EQUAL, and cursor-set returned nil, so did get-instances-by-value. i wonder why such stuff is not in tests, it seems only numbers and strings are tested as index keys. From killerstorm at newmail.ru Fri Jan 18 11:56:53 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Fri, 18 Jan 2008 13:56:53 +0200 Subject: [elephant-devel] Re: function calls as keyforms for btree-indexes References: <5bef28df0801100737t20a0edbao11db167e9b691ad6@mail.gmail.com><1200065573.3783.129.camel@penguin.yourdomain.com><5bef28df0801110834i4320e8dajc6459ab3d4e16bce@mail.gmail.com><5bef28df0801140550w103dd6aevebd3fb8b222c48ca@mail.gmail.com> <1200340878.3783.284.camel@penguin.yourdomain.com> Message-ID: SR>> In order to get the elephant tests to run (in Lispworks) I was force SR>> to alter the accessors of the Message class to not be in the keyword package, I SR>> have attached the patch for this as well. RLR> Thank you. I was green under postmodern and cl-sql on postgres under RLR> SBCL, which is quite strange, unicodepositiontest still was using :vl accessor from keyword package, and it didn't work. i've already made patch for this, it will be included in upcoming batch, so no need to worry about that From joubert at joubster.com Mon Jan 21 02:32:37 2008 From: joubert at joubster.com (Joubert Nel) Date: Sun, 20 Jan 2008 21:32:37 -0500 Subject: [elephant-devel] Getting count of the number of persistent objects of a particular class Message-ID: <1200882757.28583.66.camel@joubert-desktop> Hello, I did some reading through the mailing list archives and the Elephant manual, but cannot find a recommended way of counting the number of persistent objects of a particular class. I have a method that works (using map-class) but this is way too slow for my purposes. I did find some postings from Oct last year about this question but no definitive answer it seams. Any advice? Joubert PS: I'm using the postmodern back-end. From killerstorm at newmail.ru Mon Jan 21 08:58:33 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 21 Jan 2008 10:58:33 +0200 Subject: [elephant-devel] Re: Getting count of the number of persistent objectsof a particular class References: <1200882757.28583.66.camel@joubert-desktop> Message-ID: JN> I did some reading through the mailing list archives and the Elephant JN> manual, but cannot find a recommended way of counting the number of JN> persistent objects of a particular class. JN> I have a method that works (using map-class) but this is way too slow JN> for my purposes. JN> I did find some postings from Oct last year about this question but no JN> definitive answer it seams. JN> Any advice? as far as i know, most btree implementation do not have a fast way to count leafs -- you need to iterate through all of them. in PostgreSQL (and i think in most relational databases like that) query "SELECT COUNT(*) FROM tree11" takes time proportional to number of elements in tree11. of course it would be much faster to do "SELECT COUNT(*)" then reading all elements on Lisp side, but it will be flawed anyway. so, the advice would be to count this parameter yourself. for example, make btree class-name -> instance count. and update it in initialize-instance (you can move this functionality to some instance-counter-mixin class), and whereever you delete them.. From joubert at joubster.com Mon Jan 21 13:52:06 2008 From: joubert at joubster.com (Joubert Nel) Date: Mon, 21 Jan 2008 08:52:06 -0500 Subject: [elephant-devel] Re: Getting count of the number of persistent objectsof a particular class In-Reply-To: References: <1200882757.28583.66.camel@joubert-desktop> Message-ID: <1200923526.28583.75.camel@joubert-desktop> On Mon, 2008-01-21 at 10:58 +0200, Alex Mizrahi wrote: > JN> I did some reading through the mailing list archives and the Elephant > JN> manual, but cannot find a recommended way of counting the number of > JN> persistent objects of a particular class. > > JN> I have a method that works (using map-class) but this is way too slow > JN> for my purposes. > > JN> I did find some postings from Oct last year about this question but no > JN> definitive answer it seams. > > JN> Any advice? > > as far as i know, most btree implementation do not have a fast way to count > leafs -- you need to iterate through all of them. > in PostgreSQL (and i think in most relational databases like that) query > "SELECT COUNT(*) FROM tree11" takes time proportional to number of elements > in tree11. > of course it would be much faster to do "SELECT COUNT(*)" then reading all > elements on Lisp side, but it will be flawed anyway. Right, I'm used to doing "SELECT COUNT" SQL statements, which could be on any kind of joins and is *very* fast in a relational database. > > so, the advice would be to count this parameter yourself. for example, make > btree class-name -> instance count. and update it in initialize-instance > (you can move this functionality to some instance-counter-mixin class), and > whereever you delete them.. This is a great suggestion. Just to verify: the writes to the database from the initialize-instance method; will they occur atomically with the consing of the persistent class (i.e. I don't need to explicitly wrap things into a transaction)? Joubert From eslick at csail.mit.edu Mon Jan 21 14:54:00 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 21 Jan 2008 09:54:00 -0500 Subject: [elephant-devel] Re: Getting count of the number of persistent objectsof a particular class In-Reply-To: <1200923526.28583.75.camel@joubert-desktop> References: <1200882757.28583.66.camel@joubert-desktop> <1200923526.28583.75.camel@joubert-desktop> Message-ID: In general, instance initialization should always happen inside a transaction. This will catch any updates you do to indices or other variables. The instance initialization code in Elephant only forces transactions for subsets of initialization that use cursors, and which under BDB would lead to deadlocks or other errors. This allows developers flexibility in breaking up large initialization transactions but the consequence is that the set of side effects of a default call to make- instance are not atomic unless wrapped in a transaction. Ian On Jan 21, 2008, at 8:52 AM, Joubert Nel wrote: > > On Mon, 2008-01-21 at 10:58 +0200, Alex Mizrahi wrote: >> JN> I did some reading through the mailing list archives and the >> Elephant >> JN> manual, but cannot find a recommended way of counting the >> number of >> JN> persistent objects of a particular class. >> >> JN> I have a method that works (using map-class) but this is way >> too slow >> JN> for my purposes. >> >> JN> I did find some postings from Oct last year about this question >> but no >> JN> definitive answer it seams. >> >> JN> Any advice? >> >> as far as i know, most btree implementation do not have a fast way >> to count >> leafs -- you need to iterate through all of them. >> in PostgreSQL (and i think in most relational databases like that) >> query >> "SELECT COUNT(*) FROM tree11" takes time proportional to number of >> elements >> in tree11. >> of course it would be much faster to do "SELECT COUNT(*)" then >> reading all >> elements on Lisp side, but it will be flawed anyway. > > Right, I'm used to doing "SELECT COUNT" SQL statements, which could be > on any kind of joins and is *very* fast in a relational database. > >> >> so, the advice would be to count this parameter yourself. for >> example, make >> btree class-name -> instance count. and update it in initialize- >> instance >> (you can move this functionality to some instance-counter-mixin >> class), and >> whereever you delete them.. > > This is a great suggestion. > Just to verify: the writes to the database from the initialize- > instance > method; will they occur atomically with the consing of the persistent > class (i.e. I don't need to explicitly wrap things into a > transaction)? > > Joubert > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From joubert at joubster.com Mon Jan 21 15:29:10 2008 From: joubert at joubster.com (Joubert Nel) Date: Mon, 21 Jan 2008 10:29:10 -0500 Subject: [elephant-devel] Re: Getting count of the number of persistent objectsof a particular class In-Reply-To: References: <1200882757.28583.66.camel@joubert-desktop> <1200923526.28583.75.camel@joubert-desktop> Message-ID: <1200929350.28583.78.camel@joubert-desktop> On Mon, 2008-01-21 at 09:54 -0500, Ian Eslick wrote: > In general, instance initialization should always happen inside a > transaction. This will catch any updates you do to indices or other > variables. > > The instance initialization code in Elephant only forces transactions > for subsets of initialization that use cursors, and which under BDB > would lead to deadlocks or other errors. This allows developers > flexibility in breaking up large initialization transactions but the > consequence is that the set of side effects of a default call to make- > instance are not atomic unless wrapped in a transaction. OK, so if I add an initialize-instance :after method, for example, then a call to (make-instance 'persistent-class) must now be called from within a transaction to ensure atomicity? Joubert From eslick at csail.mit.edu Mon Jan 21 15:57:32 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 21 Jan 2008 10:57:32 -0500 Subject: [elephant-devel] Re: Getting count of the number of persistent objectsof a particular class In-Reply-To: <1200929350.28583.78.camel@joubert-desktop> References: <1200882757.28583.66.camel@joubert-desktop> <1200923526.28583.75.camel@joubert-desktop> <1200929350.28583.78.camel@joubert-desktop> Message-ID: <4C5D1AF0-2959-49A5-9F61-9254A878FA05@csail.mit.edu> If the directions are not clear, I apologize, but it is good practice to always call (make-instance 'persistent-class) from within a transaction to ensure ACID properties for object creation. This is not terribly important for interactive, non-critical uses of the DB. But in multi-threaded environments it becomes important to ensure you don't corrupt the database in odd ways. During instance creation the following subtasks are guaranteed to be in a transaction (i.e. wrapped in an ensure-transaction call): - initialization of persistent slot values - class index update for indexed classes Any other before/after/around methods on initialize-instance or shared- initialize are not wrapped in a transaction. The bottom line is that to ensure that these methods plus the two above subtasks are transactional, be sure that make-instance is called within a transaction. If you don't, you will lose Atomicity and Consistency for object creation of indexed instances or instances with any other persistent side effects in after/before/around methods. Why don't we put this in a transaction by default? In part this is tied up in the recent discussions about initialization vs. reinitialization during serialization, part is tied up in not creating a false sense of security leading to problems when you engage in natural uses of the MOP, for instance (defun initialize- instance :after ((obj my-class) ...)), and part not having been fully thought through yet. Ian On Jan 21, 2008, at 10:29 AM, Joubert Nel wrote: > > On Mon, 2008-01-21 at 09:54 -0500, Ian Eslick wrote: >> In general, instance initialization should always happen inside a >> transaction. This will catch any updates you do to indices or other >> variables. >> >> The instance initialization code in Elephant only forces transactions >> for subsets of initialization that use cursors, and which under BDB >> would lead to deadlocks or other errors. This allows developers >> flexibility in breaking up large initialization transactions but the >> consequence is that the set of side effects of a default call to >> make- >> instance are not atomic unless wrapped in a transaction. > > OK, so if I add an initialize-instance :after method, for example, > then > a call to (make-instance 'persistent-class) must now be called from > within a transaction to ensure atomicity? > > > Joubert > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Mon Jan 21 17:09:57 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 21 Jan 2008 19:09:57 +0200 Subject: [elephant-devel] Re: Getting count of the number of persistentobjectsof a particular class References: <1200882757.28583.66.camel@joubert-desktop> <1200923526.28583.75.camel@joubert-desktop> Message-ID: ??>> tree11. of course it would be much faster to do "SELECT COUNT(*)" then ??>> reading all elements on Lisp side, but it will be flawed anyway. JN> Right, I'm used to doing "SELECT COUNT" SQL statements, which could be JN> on any kind of joins and is *very* fast in a relational database. no, they are not *very* fast. SELECT COUNT(*) takes about 10 ms on a table with 10000 element in PostgreSQL. that's about 100 times slower than simple query using index. you won't notice this if you don't call it frequently, but it can be a problem for huge databases. JN> This is a great suggestion. JN> Just to verify: the writes to the database from the initialize-instance JN> method; will they occur atomically with the consing of the persistent JN> class (i.e. I don't need to explicitly wrap things into a transaction)? there is no implicit transaction around make-instance, and I would strongly advice wrapping all code that works with DB into explicit transaction anyway. From eslick at csail.mit.edu Mon Jan 21 18:26:08 2008 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 21 Jan 2008 13:26:08 -0500 Subject: [elephant-devel] Getting count of the number of persistent objects of a particular class In-Reply-To: <1200882757.28583.66.camel@joubert-desktop> References: <1200882757.28583.66.camel@joubert-desktop> Message-ID: <616B20AA-9139-4A70-B56E-13D1FC2FA439@csail.mit.edu> Part of the problem with Elephant is that we currently create CLOS objects in memory for every 'row' that is touched in a query (or touched during an index operation). We have to de-serialize the record, create an object, initialize any transient slots, etc. The SQL engine, by contrast, can just keep track of a list of OIDs and then count the size of the arrays that result from any joins. This is a much cheaper set of operations per-row (i.e. per object). If we only read OIDs into memory until the final step when we need to access a slot, we can save quite a bit of the cost of object creation. My hope is that a query system would implement this model properly. Unfortunately, this will not happen soon, as much as I'm itching to do it! However, the recent change to split object instantiation into initial and deserialization pathways should have sped things up a bit (e.g. no p-slot accesses during de-serialization). Ian On Jan 20, 2008, at 9:32 PM, Joubert Nel wrote: > Hello, > > I did some reading through the mailing list archives and the Elephant > manual, but cannot find a recommended way of counting the number of > persistent objects of a particular class. > > I have a method that works (using map-class) but this is way too slow > for my purposes. > > I did find some postings from Oct last year about this question but no > definitive answer it seams. > > Any advice? > > Joubert > > PS: I'm using the postmodern back-end. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Mon Jan 21 19:52:19 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 21 Jan 2008 21:52:19 +0200 Subject: [elephant-devel] Re: Getting count of the number of persistentobjects of a particular class References: <1200882757.28583.66.camel@joubert-desktop> <616B20AA-9139-4A70-B56E-13D1FC2FA439@csail.mit.edu> Message-ID: IE> Part of the problem with Elephant is that we currently create CLOS IE> objects in memory for every 'row' that is touched in a query (or IE> touched during an index operation). We have to de-serialize the IE> record, create an object, initialize any transient slots, etc. The IE> SQL engine, by contrast, can just keep track of a list of OIDs and IE> then count the size of the arrays that result from any joins. This is IE> a much cheaper set of operations per-row (i.e. per object). If we IE> only read OIDs into memory until the final step when we need to access IE> a slot, we can save quite a bit of the cost of object creation. also backends which connect to database via a socket have considerable communication overhead, i think it's even larger than deserialization impact. IE> My hope is that a query system would implement this model properly. IE> Unfortunately, this will not happen soon, as much as I'm itching to do IE> it! OTOH it's fairly easy to make function that counts total number of entries in btree or in some range, at least it's easy with db-postmodern -- it just translates to COUNT(*) query. if there is such a big demand for this feature, we can just implement it as backend-specific advantage ^__^. From luismbo at gmail.com Mon Jan 21 20:22:48 2008 From: luismbo at gmail.com (Luis Oliveira) Date: Mon, 21 Jan 2008 20:22:48 +0000 Subject: [elephant-devel] Re: Trouble compiling Elephant on SBCL References: <652C6C0B-2CBB-46C2-9D93-C95BFE99FE9F@csail.mit.edu> <87k5mkhed9.fsf@kzn.homelinux.org> Message-ID: Anton Kazennikov writes: > Hmm.. I've got this problem when compiling under slime with load-system. > When I do this from prompt using (asdf:oos 'asdf:load-op :elephant) the > problem is gone. Such strange heuristic ;-) Thanks everyone for the replies. This indeed turns out to be an SBCL/SLIME bug, AFAICT. I'm digging a little deeper right now and then I'll report this to slime-devel. -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ From Blog at common-lisp.net Tue Jan 22 08:16:26 2008 From: Blog at common-lisp.net (Blog at common-lisp.net) Date: 22 Jan 2008 00:16:26 -0800 Subject: [elephant-devel] How would you like to have your ad on 2 Million Websites ? Message-ID: <20080122001625.956251F2B129CAB8@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From xd at maxexp.com Wed Jan 23 06:08:43 2008 From: xd at maxexp.com (Jason Anderson) Date: Tue, 22 Jan 2008 22:08:43 -0800 Subject: [elephant-devel] bdb replication Message-ID: <5c2dc04a0801222208q3df3dc8fk2fd2d67f5169974c@mail.gmail.com> Hey all, Has anyone tried to get bdb replication working with elephant? If so, were the results successful? Thanks in advance, Jason From Feed at common-lisp.net Thu Jan 24 00:37:46 2008 From: Feed at common-lisp.net (Feed at common-lisp.net) Date: 23 Jan 2008 16:37:46 -0800 Subject: [elephant-devel] Receive hundreds of targeted hits to your website every day from the links in the feeds! Message-ID: <20080123163745.DAE7BAEEB8677A8A@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: From read at robertlread.net Thu Jan 24 02:57:05 2008 From: read at robertlread.net (Robert L. Read) Date: Wed, 23 Jan 2008 20:57:05 -0600 Subject: [elephant-devel] I have banned the recent spammers.... Message-ID: <1201143425.4276.184.camel@penguin.yourdomain.com> I am not entirely sure how to prevent this in the future. From leslie.polzer at gmx.net Thu Jan 24 08:23:21 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Thu, 24 Jan 2008 09:23:21 +0100 (CET) Subject: [elephant-devel] Schema evolution protocol Message-ID: <63714.88.73.236.141.1201163001.squirrel@mail.stardawn.org> Yesterday I came across this thing: http://common-lisp.net/project/cl-migrations/ It might be useful later on, when we start to tackle the user-managed parts of schema evolution. Ian, do you have more details on when you will be ready? Leslie -- My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Fri Jan 25 10:53:34 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 25 Jan 2008 11:53:34 +0100 (CET) Subject: [elephant-devel] Accessing a DB from multiple processes Message-ID: <63396.88.73.252.27.1201258414.squirrel@mail.stardawn.org> I know that Elephant is thread-safe. Am I right in that I can also open a store from separate Lisp images? Leslie -- My personal blog: http://blog.viridian-project.de/ From henrik at evahjelte.com Fri Jan 25 11:14:01 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 25 Jan 2008 12:14:01 +0100 Subject: [elephant-devel] Accessing a DB from multiple processes In-Reply-To: <63396.88.73.252.27.1201258414.squirrel@mail.stardawn.org> References: <63396.88.73.252.27.1201258414.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60801250314l6ac04519ha277225a4102c786@mail.gmail.com> On Jan 25, 2008 11:53 AM, Leslie P. Polzer wrote: > > I know that Elephant is thread-safe. Am I right in that I can also > open a store from separate Lisp images? The postmodern backend is almost optimized for this use case, with its globally synced cache. clsql backend works as well. bdb backend should work... I had some problems with BDB in this use case, but that was some time ago so I don't know the current state. Note that multiple image access is not tested in the elephant testcases. There is code for testing this in the project grand-prix on common-lisp.net /Henrik From joubert at joubster.com Sat Jan 26 12:57:43 2008 From: joubert at joubster.com (Joubert Nel) Date: Sat, 26 Jan 2008 07:57:43 -0500 Subject: [elephant-devel] Re: Getting count of the number of persistentobjects of a particular class In-Reply-To: References: <1200882757.28583.66.camel@joubert-desktop> <616B20AA-9139-4A70-B56E-13D1FC2FA439@csail.mit.edu> Message-ID: <1201352263.11824.8.camel@joubert-desktop> On Mon, 2008-01-21 at 21:52 +0200, Alex Mizrahi wrote: > IE> Part of the problem with Elephant is that we currently create CLOS > IE> objects in memory for every 'row' that is touched in a query (or > IE> touched during an index operation). We have to de-serialize the > IE> record, create an object, initialize any transient slots, etc. The > IE> SQL engine, by contrast, can just keep track of a list of OIDs and > IE> then count the size of the arrays that result from any joins. This is > IE> a much cheaper set of operations per-row (i.e. per object). If we > IE> only read OIDs into memory until the final step when we need to access > IE> a slot, we can save quite a bit of the cost of object creation. > > also backends which connect to database via a socket have considerable > communication overhead, i think it's even larger than deserialization > impact. > > IE> My hope is that a query system would implement this model properly. > IE> Unfortunately, this will not happen soon, as much as I'm itching to do > IE> it! > > OTOH it's fairly easy to make function that counts total number of entries > in btree or in some range, at least it's easy with db-postmodern -- it just > translates to COUNT(*) query. > if there is such a big demand for this feature, we can just implement it as > backend-specific advantage ^__^. I'm not sure how many people desire such a function, but I think it is certainly useful and something I want to leverage. I seems only BDB would not support this... Joubert From Hit-Booster at common-lisp.net Thu Jan 31 22:51:13 2008 From: Hit-Booster at common-lisp.net (Hit-Booster at common-lisp.net) Date: 31 Jan 2008 14:51:13 -0800 Subject: [elephant-devel] How to get free quality visitors to your website? Message-ID: <20080131145112.5E406F5B4D9BD458@from.header.has.no.domain> An HTML attachment was scrubbed... URL: -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Unsubscribe email.txt URL: