From henrik at evahjelte.com Thu Nov 1 17:28:53 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 1 Nov 2007 18:28:53 +0100 Subject: [elephant-devel] Schema evolution In-Reply-To: <67D83578-80F5-41DB-8BDB-39C69D57180E@csail.mit.edu> References: <67D83578-80F5-41DB-8BDB-39C69D57180E@csail.mit.edu> Message-ID: <50e8e4f60711011028h352c6091w47ae07ea40e89fc@mail.gmail.com> I am no expert at how elephant does this now, I've been digging more in the low level areas. But I think that after some time you really will want schema changes to be saved, and to be able to have different versions of classes, and to have a lazy update of instances. It makes working with the system much more secure and easier, this is probably one of the top features to ask for. Is this really so hard to do? I don't think so. About the update interface, Rucksack has a method, I have pasted it below. You could even store a little function (the source code) in the database, to be evaluated/executed when updating old versions. That way you would never be afraid to lose it somewhere in the source code. Of course, triggering updates of all obsolete instances should also be possible, that is trivial once the versioning is in place. I am willing to help with adding a feature like this, but don't have time to do it myself right now. Something for a student/ Lisp gardener? /Henrik Hjelte ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;;; Updating persistent instances ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; When a persistent object must be loaded from disk, Rucksack loads the ;; schema nr and finds the corresponding schema. If the schema is obsolete ;; (i.e. there is a schema for the same class with a higher version number), ;; Rucksack calls the generic function UPDATE-PERSISTENT-INSTANCE-FOR-REDEFINED-CLASS ;; after calling ALLOCATE-INSTANCE for the current class version. The generic ;; function is very similar to UPDATE-INSTANCE-FOR-REDEFINED-CLASS: it takes a ;; list of added slots, a list of deleted slots and a property list containing ;; the slot names and values for slots that were discarded and had values. (defgeneric update-persistent-instance-for-redefined-class (instance added-slots discarded-slots property-list &key) (:method ((instance persistent-object) added-slots discarded-slots plist &key) From franks-muc at web.de Sun Nov 4 17:01:42 2007 From: franks-muc at web.de (Frank Schorr) Date: Sun, 04 Nov 2007 18:01:42 +0100 Subject: [elephant-devel] error at delivery on LispWorks Message-ID: <789467406@web.de> When delivering my application with Lispworks on win32 at delivery level 2, the delivery process produces an error. For example: Error: An error of type CONDITIONS:ARITHMETIC-TYPE-ERROR occured, arguments : (:EXPECTED-TYPE REAL :DATUM ELEPHANT:PRIMARY :OPERATION < :OPERANDS (2 ELEPHANT:PRIMARY)) This error is not exactly reproducible. "ELEPHANT:PRIMARY" can be replaced by one or the other slot name of my persistent class. But it generates always a CONDITIONS:ARITHMETIC-TYPE-ERROR. Complete output is reproduced below. Do you have an idea where to start solving this ? At delivery levels 0 and 1, a perfectly working executable file is produced, having a size of 40MB. It should be possible to reduce this size at higher levels. Best regards, Frank Schorr Shaking stage : Shaking (1) Generation 0: Total Size 4358K, Allocated 349K, Free 4000K Generation 1: Total Size 4982K, Allocated 2196K, Free 2765K Generation 2: Total Size 25604K, Allocated 12112K, Free 13475K Generation 3: Total Size 18263K, Allocated 8611K, Free 9643K Total Size 53504K, Allocated 23269K, Free 29885K Broke pointers from 2982 gfs Number of symbols 29616 (keywords 2742 fbound 15687 bound 4070 uninterned 677) Number of classes 966 Unwinding shaking-action "?SYSTEM::ACTION-TUPLE-SYM" Unwinding shaking-action "?SYSTEM::ACTION-ITEM-OWNER" Unwinding shaking-action "?TYPE::CLEAR-TYPE-EXPANSION-CACHE" Unwinding shaking-action "?SYSTEM::*LINE-ARGUMENT-ACTIONS*" Unwinding shaking-action "?CLOS::*ISL-TABLE*" Unwinding shaking-action "?CLOS::CLASS-SHARED-INITIALIZE" Error: An error of type CONDITIONS:ARITHMETIC-TYPE-ERROR occured, arguments : (:EXPECTED-TYPE REAL :DATUM ELEPHANT:PRIMARY :OPERATION < :OPERANDS (2 ELEPHANT:PRIMARY)) Backtrace: # Call to "GET-CALL-FRAME" "FRAME-POINTER" : NIL "PREV-FRAME" : NIL Binding frame: *DEBUGGER-HOOK* : NIL SYSTEM::*INSIDE-RUN-BUILD-SCRIPT-DEBUGGER-HOOK* : NIL Call to "RUN-BUILD-SCRIPT-DEBUGGER-HOOK" (offset 145) "CC" : # Catch frame: CONDITIONS::NEW-IGNORE-ERRORS-HANDLER Binding frame: MP:*CURRENT-PROCESS* : NIL Call to "DEBUG2" (offset 3141) "CONDITION" : # "PROCESS" : NIL "SKIP-TO" : NIL Tag environment contour: Function environment contour Block environment contour: Variable environment contour: () Call to "DEBUG1" (offset 215) "(DATUM NIL)" : # "ARGUMENTS" : NIL Binding frame: MP::*PROCESSING-INTERRUPTS* : NIL *EVALHOOK* : NIL Call to "INVOKE-DEBUGGER" (offset 111) "CONDITION" : # Binding frame: CONDITIONS::*BROKEN-ON-SIGNALS* : NIL Call to "CONDITIONS-ERROR" (offset 356) "DATUM" : CONDITIONS:ARITHMETIC-TYPE-ERROR "ARGUMENTS" : (:EXPECTED-TYPE REAL :DATUM ELEPHANT:PRIMARY :OPERATION < :OPERANDS (2 ELEPHANT:PRIMARY)) Catch frame: (NIL) Catch frame: (NIL) Call to "REPORT-THE-BINARY-FN-ERROR" (offset 329) "FN-NAME" : < "ARG1" : 2 "ARG2" : ELEPHANT:PRIMARY "TYPE" : REAL "ARG1-IS-BAD" : NIL "PROMPT" : "Supply a new second argument." Call to "ARGS-TO-BINARY-ARITHMETIC-FN-NOT-OF-TYPE" (offset 197) "FN-NAME" : < "ARG1" : 2 "ARG2" : ELEPHANT:PRIMARY "TYPE" : REAL Call to "MERGE-LISTS*" (offset 87) "LIST-1" : ((ELEPHANT:PRIMARY # (:PRIMARY))) "LIST-2" : ((2 # NIL)) "PRED" : # "KEY" : # Call to "SORT-LIST" (offset 279) "LIST" : ((1 # (:DBCONNECTION-SPEC-PST)) (0 # (:FROM-OID))) "PRED" : # "KEY" : # Call to "(METHOD COMPUTE-CLASS-SHARED-INITIALIZE-INFO (T T))" (offset 258) "CLASS" : # "JUST-SLOTS-P" : T Call to "(METHOD COMPUTE-CLASS-SHARED-INITIALIZE T . NIL)" (offset 56) "CLASS" : # Call to "INSTALL-ALL-CLASS-SHARED-INITIALIZE" (offset 138) "NAME" : DB-BDB::BDB-BTREE-INDEX "CLASS" : # "METACLASSES-KEEP" : :DONT-KNOW "KEEP" : :DONT-KNOW Catch frame: # Call to "MAPHASH" (offset 356) "FUNCTION" : # "HASH-TABLE" : # Call to "INSTALL-ALL-CLASS-SHARED-INITIALIZE" (offset 90) "KEEP" : NIL "METACLASSES-KEEP" : (# # #) Call to "UNWIND-SHAKING-ACTIONS" (offset 456) "TESTING-ONLY" : NIL Binding frame: SYSTEM::*BUILT-IN-CLASSES* : NIL Catch frame: # Binding frame: *PACKAGES-FOR-WARN-ON-REDEFINITION* : ("AMD64" "ATSUI" "CAPI" "CAPI-COCOA-LIBRARY" "CAPI-INTERNALS" "CAPI-LAYOUT" "CAPI-LIBRARY" "CAPI-MOTIF-LIBRARY" "CAPI-POSTSCRIPT-LIBRARY" "CAPI-TOOLKIT" "CAPI-WIN32-LIB" "CARBON" "CLIM" "CLIM-DEFSYSTEM" "CLIM-INTERNALS" "CLIM-LISP" "CLIM-SILICA" "CLIM-SYS" "CLIM-UTILS" "CLOG-VAR" "CLOS" "COCOA" "COLOR" "COM" "COMM" "COMMON-LISP" "COMMON-PROLOG" "COMMON-UTILITIES" "COMPILER" "CONDITIONS" "CONSTANTS" "CORBA" "CORBA-INTERNAL" "CORE-GRAPHICS" "COSNAMING" "DBG" "DELIVERY" "DSPEC" "EDITOR" "ENVIRONMENT" "ENVIRONMENT-INTERNALS" "EXE-PARSER" "EXTERNAL-FORMAT" "FLI" "FLI-INTERNALS" "FOREIGN" "FOREIGN-PARSER" "GENERATOR" "GRAPHICS-PORTS" "HARLEQUIN-COMMON-LISP" "HARP" "HQN-WEB" "INS" "INTERFACE-BUILDER" "KEYWORD" "KW" "KW-TOOLS" "LISPWORKS" "LISPWORKS-TOOLS" "LOOP" "LOW64" "LW-XP" "MAIL" "MATCH" "MP" "OBJC" "ODBC-COMMON" "OMG.ORG/OPERATION" "OMG.ORG/PORTABLESERVER" "OMG.ORG/ROOT" "PARSERGEN" "PC386" "PKG" "POSTGRESQL" "POSTSCRIPT-CLIM" "RAW" "RC-PARSER" "REG" "RS6K" "RUNTIME" "SAVED-STUFF" "SCM" "SERIAL-PORT" "SETF" "SHELL" "SNAKE" "SQL" "SQL-COMMON" "STEPPER" "STREAM" "STRUCTURE" "SYSTEM" "TRANS" "TYPE" "WALKER" "WIN32" "WIN32-LIB-CLIM" "WSPARC" "X-INTERNALS" "X-LIBRARY" "X-TOOLS" "X-UTILITIES" "XM-INTERNALS" "XM-LIB-CLIM" "XM-LIBRARY" "XMA-LIBRARY" "XT-INTERNALS" "XT-LIBRARY") Binding frame: SYSTEM::*SHAKE-CLASS-DIRECT-METHODS* : NIL Binding frame: SYSTEM::*SHAKE-CLASS-ACCESSORS* : NIL Binding frame: SYSTEM::*SHAKE-CLASSES* : NIL Call to "DELIVER-SHAKE" (offset 2655) Catch frame: # Binding frame: SYSTEM::*CLEANING* : NIL Call to "DELIVER-FROM-MAIN" (offset 371) Catch frame: SYSTEM::EXIT-LISPWORKS Catch frame: SYSTEM::START-UP Catch frame: (NIL) Call to "START-FUNCTION" (offset 116) "GC-MESSAGES" : # "START-FUNCTION" Quitting _________________________________________________________________________ Mit der Gruppen-SMS von WEB.DE FreeMail k?nnen Sie eine SMS an alle Freunde gleichzeitig schicken: http://freemail.web.de/features/?mc=021179 From eslick at csail.mit.edu Wed Nov 7 22:28:51 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 7 Nov 2007 17:28:51 -0500 Subject: [elephant-devel] error at delivery on LispWorks In-Reply-To: <789467406@web.de> References: <789467406@web.de> Message-ID: I don't have a full lispworks implementation so I probably can't debug this myself. However its choking on btree-index which is a class instance of the persistent-metaclass. This may be due to something missing in the mop support for lispworks. It looks like a persistent slot record is providing a representation (slotname value initarg) whereas the delivery procedure compute-class- shared-initialize is expecting the computed (effective?) representation which is (offset-integer value initarg). I would look at what the lispworks mop expects as values from some of the slot metaclass functions. You might start by figuring out where in metaclass.lisp or classes.lisp the data structure (ELEPHANT:PRIMARY # (:PRIMARY)) is generated. Good luck! Ian What I think is happening is that the procedure which On Nov 4, 2007, at 12:01 PM, Frank Schorr wrote: > When delivering my application > with Lispworks on win32 at delivery level 2, the > delivery process produces an error. For example: > > Error: An error of type CONDITIONS:ARITHMETIC-TYPE-ERROR occured, > arguments : (:EXPECTED-TYPE REAL :DATUM ELEPHANT:PRIMARY :OPERATION > < :OPERANDS (2 ELEPHANT:PRIMARY)) > > This error is not exactly reproducible. "ELEPHANT:PRIMARY" can be > replaced by one or the other > slot name of my persistent class. But it generates always a > CONDITIONS:ARITHMETIC-TYPE-ERROR. > Complete output is reproduced below. > > Do you have an idea where to start solving this ? > > At delivery levels 0 and 1, a perfectly working executable file is > produced, > having a size of 40MB. It should be possible to reduce this size at > higher levels. > > Best regards, > > Frank Schorr > > Shaking stage : Shaking (1) > Generation 0: Total Size 4358K, Allocated 349K, Free 4000K > Generation 1: Total Size 4982K, Allocated 2196K, Free 2765K > Generation 2: Total Size 25604K, Allocated 12112K, Free 13475K > Generation 3: Total Size 18263K, Allocated 8611K, Free 9643K > > Total Size 53504K, Allocated 23269K, Free 29885K > Broke pointers from 2982 gfs > > Number of symbols 29616 (keywords 2742 fbound 15687 bound 4070 > uninterned 677) > > Number of classes 966 > > > Unwinding shaking-action "?SYSTEM::ACTION-TUPLE-SYM" > > Unwinding shaking-action "?SYSTEM::ACTION-ITEM-OWNER" > > Unwinding shaking-action "?TYPE::CLEAR-TYPE-EXPANSION-CACHE" > > Unwinding shaking-action "?SYSTEM::*LINE-ARGUMENT-ACTIONS*" > > Unwinding shaking-action "?CLOS::*ISL-TABLE*" > > Unwinding shaking-action "?CLOS::CLASS-SHARED-INITIALIZE" > Error: An error of type CONDITIONS:ARITHMETIC-TYPE-ERROR occured, > arguments : (:EXPECTED-TYPE REAL :DATUM ELEPHANT:PRIMARY :OPERATION > < :OPERANDS (2 ELEPHANT:PRIMARY)) > > Backtrace: > # > > > Call to "GET-CALL-FRAME" > "FRAME-POINTER" : NIL > "PREV-FRAME" : NIL > > Binding frame: > *DEBUGGER-HOOK* : NIL > SYSTEM::*INSIDE-RUN-BUILD-SCRIPT-DEBUGGER-HOOK* : NIL > > Call to "RUN-BUILD-SCRIPT-DEBUGGER-HOOK" (offset 145) > "CC" : # > > Catch frame: CONDITIONS::NEW-IGNORE-ERRORS-HANDLER > > Binding frame: > MP:*CURRENT-PROCESS* : NIL > > Call to "DEBUG2" (offset 3141) > "CONDITION" : # > "PROCESS" : NIL > "SKIP-TO" : NIL > > Tag environment contour: > Function environment contour > Block environment contour: > Variable environment contour: () > Call to "DEBUG1" (offset 215) > "(DATUM NIL)" : # > "ARGUMENTS" : NIL > > Binding frame: > MP::*PROCESSING-INTERRUPTS* : NIL > *EVALHOOK* : NIL > > Call to "INVOKE-DEBUGGER" (offset 111) > "CONDITION" : # > > Binding frame: > CONDITIONS::*BROKEN-ON-SIGNALS* : NIL > > Call to "CONDITIONS-ERROR" (offset 356) > "DATUM" : CONDITIONS:ARITHMETIC-TYPE-ERROR > "ARGUMENTS" : (:EXPECTED-TYPE REAL :DATUM > ELEPHANT:PRIMARY :OPERATION < :OPERANDS (2 ELEPHANT:PRIMARY)) > > Catch frame: (NIL) > > Catch frame: (NIL) > > Call to "REPORT-THE-BINARY-FN-ERROR" (offset 329) > "FN-NAME" : < > "ARG1" : 2 > "ARG2" : ELEPHANT:PRIMARY > "TYPE" : REAL > "ARG1-IS-BAD" : NIL > "PROMPT" : "Supply a new second argument." > > Call to "ARGS-TO-BINARY-ARITHMETIC-FN-NOT-OF-TYPE" (offset 197) > "FN-NAME" : < > "ARG1" : 2 > "ARG2" : ELEPHANT:PRIMARY > "TYPE" : REAL > > Call to "MERGE-LISTS*" (offset 87) > "LIST-1" : ((ELEPHANT:PRIMARY # (:PRIMARY))) > "LIST-2" : ((2 # NIL)) > "PRED" : # > "KEY" : # > > Call to "SORT-LIST" (offset 279) > "LIST" : ((1 # (:DBCONNECTION-SPEC-PST)) (0 > # (:FROM-OID))) > "PRED" : # > "KEY" : # > > Call to "(METHOD COMPUTE-CLASS-SHARED-INITIALIZE-INFO (T > T))" (offset 258) > "CLASS" : # INDEX 21D0254B> > "JUST-SLOTS-P" : T > > Call to "(METHOD COMPUTE-CLASS-SHARED-INITIALIZE T . NIL)" (offset > 56) > "CLASS" : # 21D0254B> > > Call to "INSTALL-ALL-CLASS-SHARED-INITIALIZE" (offset 138) > "NAME" : DB-BDB::BDB-BTREE-INDEX > "CLASS" : # BTREE-INDEX 21D0254B> > "METACLASSES-KEEP" : :DONT-KNOW > "KEEP" : :DONT-KNOW > > Catch frame: # > > Call to "MAPHASH" (offset 356) > "FUNCTION" : # 20269642> > "HASH-TABLE" : # > > Call to "INSTALL-ALL-CLASS-SHARED-INITIALIZE" (offset 90) > "KEEP" : NIL > "METACLASSES-KEEP" : (# > # # FUNCALLABLE-STANDARD-CLASS 2095033F>) > > Call to "UNWIND-SHAKING-ACTIONS" (offset 456) > "TESTING-ONLY" : NIL > > Binding frame: > SYSTEM::*BUILT-IN-CLASSES* : NIL > > Catch frame: # > > Binding frame: > *PACKAGES-FOR-WARN-ON-REDEFINITION* : ("AMD64" "ATSUI" "CAPI" "CAPI- > COCOA-LIBRARY" "CAPI-INTERNALS" "CAPI-LAYOUT" "CAPI-LIBRARY" "CAPI- > MOTIF-LIBRARY" "CAPI-POSTSCRIPT-LIBRARY" "CAPI-TOOLKIT" "CAPI-WIN32- > LIB" "CARBON" "CLIM" "CLIM-DEFSYSTEM" "CLIM-INTERNALS" "CLIM-LISP" > "CLIM-SILICA" "CLIM-SYS" "CLIM-UTILS" "CLOG-VAR" "CLOS" "COCOA" > "COLOR" "COM" "COMM" "COMMON-LISP" "COMMON-PROLOG" "COMMON- > UTILITIES" "COMPILER" "CONDITIONS" "CONSTANTS" "CORBA" "CORBA- > INTERNAL" "CORE-GRAPHICS" "COSNAMING" "DBG" "DELIVERY" "DSPEC" > "EDITOR" "ENVIRONMENT" "ENVIRONMENT-INTERNALS" "EXE-PARSER" > "EXTERNAL-FORMAT" "FLI" "FLI-INTERNALS" "FOREIGN" "FOREIGN-PARSER" > "GENERATOR" "GRAPHICS-PORTS" "HARLEQUIN-COMMON-LISP" "HARP" "HQN- > WEB" "INS" "INTERFACE-BUILDER" "KEYWORD" "KW" "KW-TOOLS" "LISPWORKS" > "LISPWORKS-TOOLS" "LOOP" "LOW64" "LW-XP" "MAIL" "MATCH" "MP" "OBJC" > "ODBC-COMMON" "OMG.ORG/OPERATION" "OMG.ORG/PORTABLESERVER" "OMG.ORG/ > ROOT" "PARSERGEN" "PC386" "PKG" "POSTGRESQL" "POSTSCRIPT-CLIM" "RAW" > "RC-PARSER" "REG" "RS6K" "RUNTIME" "SAVED-STUFF" "SCM" "SERIAL-PORT" > "SETF" "SHELL" "SNAKE" "SQL" "SQL-COMMON" "STEPPER" "STREAM" > "STRUCTURE" "SYSTEM" "TRANS" "TYPE" "WALKER" "WIN32" "WIN32-LIB- > CLIM" "WSPARC" "X-INTERNALS" "X-LIBRARY" "X-TOOLS" "X-UTILITIES" "XM- > INTERNALS" "XM-LIB-CLIM" "XM-LIBRARY" "XMA-LIBRARY" "XT-INTERNALS" > "XT-LIBRARY") > > Binding frame: > SYSTEM::*SHAKE-CLASS-DIRECT-METHODS* : NIL > > Binding frame: > SYSTEM::*SHAKE-CLASS-ACCESSORS* : NIL > > Binding frame: > SYSTEM::*SHAKE-CLASSES* : NIL > > Call to "DELIVER-SHAKE" (offset 2655) > > Catch frame: # > > Binding frame: > SYSTEM::*CLEANING* : NIL > > Call to "DELIVER-FROM-MAIN" (offset 371) > > Catch frame: SYSTEM::EXIT-LISPWORKS > > Catch frame: SYSTEM::START-UP > > Catch frame: (NIL) > > Call to "START-FUNCTION" (offset 116) > "GC-MESSAGES" : # > > > "START-FUNCTION" > Quitting > _________________________________________________________________________ > Mit der Gruppen-SMS von WEB.DE FreeMail k?nnen Sie eine SMS an alle > Freunde gleichzeitig schicken: http://freemail.web.de/features/?mc=021179 > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Fri Nov 16 16:08:16 2007 From: read at robertlread.net (Robert L. Read) Date: Fri, 16 Nov 2007 10:08:16 -0600 Subject: [elephant-devel] Elephant 0.9.1 officially released (The "Postmodern" release) Message-ID: <1195229296.7772.976.camel@penguin.yourdomain.com> Elephant 0.9.1 is now officially released. Instructions for downloading it via darcs can be found here: http://common-lisp.net/project/elephant/downloads.html The release provides official support for using Postmodern http://common-lisp.net/project/postmodern/ as a back-end data repository on top of Postgres. This back-end is about three times faster than the generic SQL interface on top of Postgres. There are some other improvements as well. Note that existing Postgres repositories are not compatible with a Postmodern repository. However, Elephant provides powerful multi-repository migration. If you have an existing Postgres repository and want to migrate to Postmodern, you should create a completely new Postmodern repository and migrate all of your data into it. The migration can be accomplished with a single command. If you have been using the (unofficially released) postmodern backend, you will have to migrate your data into a CL-SQL based postgres backend, or into a BDB backend, and then migrate into your new postmodern repository. This release is not compatible with previous (development) versions of the postmodern functionality. Thanks very much to Henrik Hejte and the postmodern team for this functionality, and to Ian Eslick for other improvements. This release has been tested on a 64-bit Linux with SBCL 1.0.10, and other platforms as well. If anybody downloads it and tests it, please run the (large) set of automated tests and report your results to elphant-devel at common-lisp.net. I will update the platforms page: http://common-lisp.net/project/elephant/platforms.html as I receive your updates. Henrik has creates some new documentation which can be build from the source, but I have been unable to build it, so have not updated the website. -- Robert L. Read Konsenti.com From cmcurtis at prevmedicine.com Tue Nov 20 16:56:05 2007 From: cmcurtis at prevmedicine.com (Chris Curtis) Date: Tue, 20 Nov 2007 11:56:05 -0500 Subject: [elephant-devel] Error loading on LW5? Message-ID: <44FBED30-C6F3-48D3-BF0D-A2153A70F7CA@prevmedicine.com> Hi, all... I'm getting the following error after (asdf:oos 'asdf:load- op :elephant) on LispWorks 5 (Mac OS X): ;;; Compiling file /Users/cmcurtis/lisp/source/elephant/src/elephant/ classes.lisp ... ;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 0 ;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3 ;;; Source level debugging is on ;;; Source file recording is on ;;; Cross referencing is on ; (TOP-LEVEL-FORM 1) ; (TOP-LEVEL-FORM 2) ; (DEFVAR ELEPHANT::*DEBUG-SI*) ; (METHOD INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) **++++ Error in (DEFCLASS ELEPHANT:PERSISTENT-OBJECT): Undefined function SET-DB-SYNCH called with arguments (# :CLASS). ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-METACLASS T)) ; (METHOD CLOS:FINALIZE-INHERITANCE :AROUND (ELEPHANT:PERSISTENT- METACLASS)) ; (METHOD REINITIALIZE-INSTANCE :AROUND (ELEPHANT:PERSISTENT-METACLASS)) ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-OBJECT T)) ; ELEPHANT::INITIALIZE-PERSISTENT-SLOTS ; (SUBFUNCTION (DEFCLASS ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA)) ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) ; ELEPHANT::WARN-ABOUT-DROPPED-SLOTS ; ELEPHANT::DROP-SLOTS ; (METHOD UPDATE-INSTANCE-FOR-REDEFINED-CLASS :AROUND (ELEPHANT:PERSISTENT-OBJECT T T T)) ; (METHOD CHANGE-CLASS :AROUND (ELEPHANT:PERSISTENT STANDARD-CLASS)) ; (METHOD CHANGE-CLASS :AROUND (STANDARD-OBJECT ELEPHANT:PERSISTENT- METACLASS)) ; (METHOD UPDATE-INSTANCE-FOR-DIFFERENT-CLASS :AROUND (ELEPHANT:PERSISTENT ELEPHANT:PERSISTENT)) ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION)) ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT- METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- DEFINITION)) ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION)) ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT SYMBOL)) ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT- METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- DEFINITION)) ; ELEPHANT::VALID-PERSISTENT-REFERENCE-P ; (SUBFUNCTION (DEFCLASS ELEPHANT:CROSS-REFERENCE-ERROR) (DEFINE- CONDITION ELEPHANT:CROSS-REFERENCE-ERROR)) ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) ; ELEPHANT::SIGNAL-CROSS-REFERENCE-ERROR ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT T)) ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT- METACLASS ELEPHANT:PERSISTENT-OBJECT T)) ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT- METACLASS ELEPHANT:PERSISTENT-OBJECT T)) ; (TOP-LEVEL-FORM 3) ; *** 1 error detected, no fasl file produced. Warning: COMPILE-FILE warned while performing # on #. Warning: COMPILE-FILE failed while performing # on #. Error: erred while invoking # on # From what little I can tell, it doesn't look like classindex.lisp is getting compiled first, which is where set-db-sync appears to be defined. But that's just my quick assessment. Any ideas? --chris From eslick at csail.mit.edu Tue Nov 20 18:10:21 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 20 Nov 2007 13:10:21 -0500 Subject: [elephant-devel] Error loading on LW5? In-Reply-To: <44FBED30-C6F3-48D3-BF0D-A2153A70F7CA@prevmedicine.com> References: <44FBED30-C6F3-48D3-BF0D-A2153A70F7CA@prevmedicine.com> Message-ID: <0CCF0C27-3E57-44E5-85CF-8E42233DD1D8@csail.mit.edu> Which code version is this? On Nov 20, 2007, at 11:56 AM, Chris Curtis wrote: > Hi, all... I'm getting the following error after (asdf:oos > 'asdf:load-op :elephant) on LispWorks 5 (Mac OS X): > > ;;; Compiling file /Users/cmcurtis/lisp/source/elephant/src/elephant/ > classes.lisp ... > ;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 0 > ;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3 > ;;; Source level debugging is on > ;;; Source file recording is on > ;;; Cross referencing is on > ; (TOP-LEVEL-FORM 1) > ; (TOP-LEVEL-FORM 2) > ; (DEFVAR ELEPHANT::*DEBUG-SI*) > ; (METHOD INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) > > > **++++ Error in (DEFCLASS ELEPHANT:PERSISTENT-OBJECT): > Undefined function SET-DB-SYNCH called with arguments (# METACLASS PERSISTENT-OBJECT 21EA0CF3> :CLASS). > ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-METACLASS T)) > ; (METHOD CLOS:FINALIZE-INHERITANCE :AROUND (ELEPHANT:PERSISTENT- > METACLASS)) > ; (METHOD REINITIALIZE-INSTANCE :AROUND (ELEPHANT:PERSISTENT- > METACLASS)) > ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-OBJECT T)) > ; ELEPHANT::INITIALIZE-PERSISTENT-SLOTS > ; (SUBFUNCTION (DEFCLASS ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) > (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA)) > ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) > ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) > ; ELEPHANT::WARN-ABOUT-DROPPED-SLOTS > ; ELEPHANT::DROP-SLOTS > ; (METHOD UPDATE-INSTANCE-FOR-REDEFINED-CLASS :AROUND > (ELEPHANT:PERSISTENT-OBJECT T T T)) > ; (METHOD CHANGE-CLASS :AROUND (ELEPHANT:PERSISTENT STANDARD-CLASS)) > ; (METHOD CHANGE-CLASS :AROUND (STANDARD-OBJECT ELEPHANT:PERSISTENT- > METACLASS)) > ; (METHOD UPDATE-INSTANCE-FOR-DIFFERENT-CLASS :AROUND > (ELEPHANT:PERSISTENT ELEPHANT:PERSISTENT)) > ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS > ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION)) > ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT- > METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- > DEFINITION)) > ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT- > METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- > DEFINITION)) > ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT- > METACLASS ELEPHANT:PERSISTENT-OBJECT SYMBOL)) > ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT- > METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- > DEFINITION)) > ; ELEPHANT::VALID-PERSISTENT-REFERENCE-P > ; (SUBFUNCTION (DEFCLASS ELEPHANT:CROSS-REFERENCE-ERROR) (DEFINE- > CONDITION ELEPHANT:CROSS-REFERENCE-ERROR)) > ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) > ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) > ; ELEPHANT::SIGNAL-CROSS-REFERENCE-ERROR > ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS > ELEPHANT:PERSISTENT-OBJECT T)) > ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT- > METACLASS ELEPHANT:PERSISTENT-OBJECT T)) > ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT- > METACLASS ELEPHANT:PERSISTENT-OBJECT T)) > ; (TOP-LEVEL-FORM 3) > ; *** 1 error detected, no fasl file produced. > Warning: COMPILE-FILE warned while performing # 21399A9F> on > #. > Warning: COMPILE-FILE failed while performing # 21399A9F> on > #. > Error: erred while invoking # on > # > > > From what little I can tell, it doesn't look like classindex.lisp is > getting compiled first, which is where set-db-sync appears to be > defined. But that's just my quick assessment. > > Any ideas? > > --chris > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From cmcurtis at prevmedicine.com Tue Nov 20 20:24:15 2007 From: cmcurtis at prevmedicine.com (Chris Curtis) Date: Tue, 20 Nov 2007 15:24:15 -0500 Subject: [elephant-devel] Error loading on LW5? In-Reply-To: <0CCF0C27-3E57-44E5-85CF-8E42233DD1D8@csail.mit.edu> References: <44FBED30-C6F3-48D3-BF0D-A2153A70F7CA@prevmedicine.com> <0CCF0C27-3E57-44E5-85CF-8E42233DD1D8@csail.mit.edu> Message-ID: <47AD7436-24F0-4DCC-B67F-1CA53DF4FBD2@prevmedicine.com> Sorry...this is 0.9.1 out of darcs. --chris On Nov 20, 2007, at 1:10 PM, "Ian Eslick" wrote: > Which code version is this? > > On Nov 20, 2007, at 11:56 AM, Chris Curtis wrote: > >> Hi, all... I'm getting the following error after (asdf:oos >> 'asdf:load-op :elephant) on LispWorks 5 (Mac OS X): >> >> ;;; Compiling file /Users/cmcurtis/lisp/source/elephant/src/ >> elephant/classes.lisp ... >> ;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 0 >> ;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3 >> ;;; Source level debugging is on >> ;;; Source file recording is on >> ;;; Cross referencing is on >> ; (TOP-LEVEL-FORM 1) >> ; (TOP-LEVEL-FORM 2) >> ; (DEFVAR ELEPHANT::*DEBUG-SI*) >> ; (METHOD INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) >> >> >> **++++ Error in (DEFCLASS ELEPHANT:PERSISTENT-OBJECT): >> Undefined function SET-DB-SYNCH called with arguments (#> METACLASS PERSISTENT-OBJECT 21EA0CF3> :CLASS). >> ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-METACLASS >> T)) >> ; (METHOD CLOS:FINALIZE-INHERITANCE :AROUND (ELEPHANT:PERSISTENT- >> METACLASS)) >> ; (METHOD REINITIALIZE-INSTANCE :AROUND (ELEPHANT:PERSISTENT- >> METACLASS)) >> ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-OBJECT T)) >> ; ELEPHANT::INITIALIZE-PERSISTENT-SLOTS >> ; (SUBFUNCTION (DEFCLASS ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) >> (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA)) >> ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) >> ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) >> ; ELEPHANT::WARN-ABOUT-DROPPED-SLOTS >> ; ELEPHANT::DROP-SLOTS >> ; (METHOD UPDATE-INSTANCE-FOR-REDEFINED-CLASS :AROUND >> (ELEPHANT:PERSISTENT-OBJECT T T T)) >> ; (METHOD CHANGE-CLASS :AROUND (ELEPHANT:PERSISTENT STANDARD-CLASS)) >> ; (METHOD CHANGE-CLASS :AROUND (STANDARD-OBJECT ELEPHANT:PERSISTENT- >> METACLASS)) >> ; (METHOD UPDATE-INSTANCE-FOR-DIFFERENT-CLASS :AROUND >> (ELEPHANT:PERSISTENT ELEPHANT:PERSISTENT)) >> ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- >> DEFINITION)) >> ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- >> DEFINITION)) >> ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- >> DEFINITION)) >> ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT SYMBOL)) >> ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT- >> DEFINITION)) >> ; ELEPHANT::VALID-PERSISTENT-REFERENCE-P >> ; (SUBFUNCTION (DEFCLASS ELEPHANT:CROSS-REFERENCE-ERROR) (DEFINE- >> CONDITION ELEPHANT:CROSS-REFERENCE-ERROR)) >> ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) >> ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) >> ; ELEPHANT::SIGNAL-CROSS-REFERENCE-ERROR >> ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT T)) >> ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT T)) >> ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT- >> METACLASS ELEPHANT:PERSISTENT-OBJECT T)) >> ; (TOP-LEVEL-FORM 3) >> ; *** 1 error detected, no fasl file produced. >> Warning: COMPILE-FILE warned while performing #> 21399A9F> on >> #. >> Warning: COMPILE-FILE failed while performing #> 21399A9F> on >> #. >> Error: erred while invoking # on >> # >> >> >> From what little I can tell, it doesn't look like classindex.lisp >> is getting compiled first, which is where set-db-sync appears to be >> defined. But that's just my quick assessment. >> >> Any ideas? >> >> --chris >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Thu Nov 22 16:37:29 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 22 Nov 2007 17:37:29 +0100 Subject: [elephant-devel] Bug in latest relase Message-ID: <50e8e4f60711220837o1ca2149by4bdfe3fbec212786@mail.gmail.com> It seems a bug crept into the latest release, at least we postmodern might get into problems with a kind-hints parameter. A fix is attached as a darcs patch, with another small update. Alex Mizrahi suggested to me it must be called the release effect Elephant was staying stable for monthes before release, but release was corrupted Loosly connected to the demo effect which also happened to me today, showing off your cool product for investors and it suddently doesn't work as you want. /Henrik Hjelte Thu Nov 22 16:10:46 CET 2007 Henrik Hjelte * remove kind-hints parameter from add-index Probably a coming feature from Ian, but right now it breaks the generic function add-index and thus postmodern, so I removed it for now. Shall I send this patch? (1/2) [ynWvpxqadjk], or ? for help: y Thu Nov 22 16:19:29 CET 2007 Henrik Hjelte * removed a little compiler warning (typo) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: kindhints.patch Type: text/x-patch Size: 944 bytes Desc: not available URL: From henrik at evahjelte.com Fri Nov 23 15:00:15 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 23 Nov 2007 16:00:15 +0100 Subject: [elephant-devel] more problems Message-ID: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> Hello everyone, I have just found another problem in the latest 0.9.1 release, db-postmodern only. If anyone is using it, I recommend to darcs unpull the following patch: "postmodern removed ugly-fix from pm-btree-index and made char-columns hardcoded (removed other option)" It seems the ugly fix was needed after all, but the problem doesn't show up in the elephant testcases. The problems shows up (for me) in get-instances-by-value with strings, so you probably don't have to migrate your databases after unpulling. I'll try to come up with a better solution. /Henrik Fri Nov 16 09:36:34 CST 2007 ieslick at common-lisp.net tagged ELEPHANT-0-9-1 Shall I unpull this patch? (3/131) [ynWvpxqadjk], or ? for help: y Sun Nov 4 14:48:02 CST 2007 eslick at common-lisp.net * Fixes to enable the docs to build (on OS X / SBCL using 'make' in elephant/doc) Shall I unpull this patch? (4/131) [ynWvpxqadjk], or ? for help: n Tue Nov 6 02:02:59 CST 2007 Henrik Hjelte * a little comment update Shall I unpull this patch? (5/131) [ynWvpxqadjk], or ? for help: n Tue Nov 6 02:02:16 CST 2007 Henrik Hjelte * postmodern removed ugly-fix from pm-btree-index and made char-columns hardcoded (removed other option). Shall I unpull this patch? (6/131) [ynWvpxqadjk], or ? for help: y Thu Nov 1 09:43:20 CDT 2007 Henrik Hjelte * random test for serializer Shall I unpull this patch? (7/131) [ynWvpxqadjk], or ? for help: d -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alain.Picard at memetrics.com Wed Nov 28 08:53:18 2007 From: Alain.Picard at memetrics.com (Alain Picard) Date: Wed, 28 Nov 2007 19:53:18 +1100 Subject: [elephant-devel] Severely poor performance in some obvious cases In-Reply-To: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> (Henrik Hjelte's message of "Fri\, 23 Nov 2007 16\:00\:15 +0100") References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> Message-ID: <87hcj6n43l.fsf@memetrics.com> Dear Elephant developers, I've been considering using Elephant for a project of mine, and have been doing some basic performance tests, using the new postmodern back end (which seems way cool, btw). The scenario I'm testing is something like this; you have a base class: (defclass person-mixin () ((name :accessor person-name :initarg :name :index t)) (:metaclass persistent-metaclass)) and a derived one: (defclass employee (person-mixin) ((job :accessor job :initarg :job)) (:metaclass persistent-metaclass)) And you go off an make a million instances of employees. [Let's say we're a very big corporation. :-)] Then when I did the following: (time (get-instance-by-value 'employee 'name name)) I was surprised to find that not only is it slow, but it conses like a madman. This led me to inspect what this function actually does, and it turns out that it ends up doing a map-index, which does a with-btree-cursor to find a get-instances-by-value and then throws away all but the first. ;;; Current definition, in 0.9.1 (defmethod get-instance-by-value ((class symbol) slot-name value) (let ((list (get-instances-by-value (find-class class) slot-name value))) (when (consp list) (car list)))) (defmethod get-instance-by-value ((class persistent-metaclass) slot-name value) (let ((list (get-instances-by-value class slot-name value))) (when (consp list) (car list)))) It seems odd to create a cursor to find something when you have an index on that slot. Also, it seems to me users of GET-INSTANCE-BY-VALUE probably imagine there is only 1 instance to return; and so would there be a huge problem in using something like the following instead: ;;; Proposed definitions: (defmethod get-instance-by-value ((class persistent-metaclass) slot-name value) (let ((bt (find-inverted-index class slot-name))) (if bt (get-value value bt) ; Do it the "simple" way (first (get-instances-by-value class slot-name value))))) (defmethod get-instance-by-value ((class symbol) slot-name value) (get-instance-by-value (find-class class) slot-name value)) This is more than a factor of 10 faster under elephant/postmodern for a class with 30,000 instances. Am I missing something really basic here? Is there a simpler way to do what I want without this performance penalty? Will this simply not work for some other back ends I'm not aware of? I feel a certain tension in the code trying to be "all things to all back-ends", and certain decisions are clearly inspired by the Berkeley DB back end, which sadly I could not use for the venture I have in mind (for licensing reasons). Lastly, is there a way to trace all the SQL commands going back and forth to postgresql in postmodern? So far I've resorted to Postgres statement logging, which is painful to match up with what the application does. I'm looking for the postmodern equivalent of CLSQL's START-SQL-RECORDING. Thanks in advance! Alain Picard -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From eslick at csail.mit.edu Wed Nov 28 23:37:09 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 28 Nov 2007 18:37:09 -0500 Subject: [elephant-devel] Severely poor performance in some obvious cases In-Reply-To: <87hcj6n43l.fsf@memetrics.com> References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> Message-ID: <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> Originally I wrote it get-instance-by-value as a convenience function when I knew that there was only one object being returned by an index. If there are multiple objects that match the input term, then get-instance-by-value is effectively returning a random one. There is going to be a huge performance difference on sets of this size between the BDB backend and the original CL-SQL backend because the SQL backend cannot support the ordering constraints of an index without reading all the objects into memory O(n) vs O(s) where s is the subset of the index size n that matches the input term where s can = 1. I'm surprised you are seeing this difference on Postmodern, unless your input name is matching a large number of objects (i.e. s is large). I had thought that the native PostgreSQL backend, postmodern, fixed the linear cost problem of CL-SQL indices and provides the same O(s) performance that BDB does, but I'm not certain (Henrik? Robert?). It sounds like you are seeing a high linear cost. As you point out, when the duplicate set is large, get-value on the secondary index does exactly the same thing without having to load in the whole duplicate set. The fix you provided behaves the same and handles the case where you want a random member of a set and the set is large so there is no reason not to do it. I'll go ahead and add this patch. There is not SQL recording built into Elephant that I'm aware of, however Henrik or Robert should weigh in on this as it may be easy to add. Cheers, Ian On Nov 28, 2007, at 3:53 AM, Alain Picard wrote: > > Dear Elephant developers, > > I've been considering using Elephant for a project of mine, > and have been doing some basic performance tests, using the > new postmodern back end (which seems way cool, btw). > > The scenario I'm testing is something like this; you > have a base class: > > (defclass person-mixin () > ((name :accessor person-name :initarg :name :index t)) > (:metaclass persistent-metaclass)) > > and a derived one: > > (defclass employee (person-mixin) > ((job :accessor job > :initarg :job)) > (:metaclass persistent-metaclass)) > > And you go off an make a million instances of employees. > [Let's say we're a very big corporation. :-)] > > Then when I did the following: > > (time (get-instance-by-value 'employee 'name name)) > > I was surprised to find that not only is it slow, but it conses > like a madman. This led me to inspect what this function actually > does, and it turns out that it ends up doing a map-index, which > does a with-btree-cursor to find a get-instances-by-value and > then throws away all but the first. > > ;;; Current definition, in 0.9.1 > > (defmethod get-instance-by-value ((class symbol) slot-name value) > (let ((list (get-instances-by-value (find-class class) slot-name > value))) > (when (consp list) > (car list)))) > > (defmethod get-instance-by-value ((class persistent-metaclass) slot- > name value) > (let ((list (get-instances-by-value class slot-name value))) > (when (consp list) > (car list)))) > > It seems odd to create a cursor to find something when you have an > index on that slot. Also, it seems to me users of > GET-INSTANCE-BY-VALUE probably imagine there is only 1 instance to > return; and so would there be a huge problem in using something like > the following instead: > > ;;; Proposed definitions: > > (defmethod get-instance-by-value ((class persistent-metaclass) slot- > name value) > (let ((bt (find-inverted-index class slot-name))) > (if bt > (get-value value bt) ; Do it the "simple" way > (first (get-instances-by-value class slot-name value))))) > > (defmethod get-instance-by-value ((class symbol) slot-name value) > (get-instance-by-value (find-class class) slot-name value)) > > This is more than a factor of 10 faster under elephant/postmodern > for a class with 30,000 instances. > > Am I missing something really basic here? Is there a simpler > way to do what I want without this performance penalty? > Will this simply not work for some other back ends I'm not > aware of? I feel a certain tension in the code trying to > be "all things to all back-ends", and certain decisions are > clearly inspired by the Berkeley DB back end, which sadly I > could not use for the venture I have in mind (for licensing reasons). > > > Lastly, is there a way to trace all the SQL commands going > back and forth to postgresql in postmodern? So far I've resorted > to Postgres statement logging, which is painful to match up > with what the application does. I'm looking for the postmodern > equivalent of CLSQL's START-SQL-RECORDING. > > > Thanks in advance! > > Alain Picard > > > > -- > Please read about why Top Posting > is evil at: http://en.wikipedia.org/wiki/Top-posting > and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html > > Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Thu Nov 29 09:02:34 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 29 Nov 2007 10:02:34 +0100 Subject: [elephant-devel] Severely poor performance in some obvious cases In-Reply-To: <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> Message-ID: <50e8e4f60711290102k39102c40ucee3ec85f1a523f4@mail.gmail.com> We have recently discovered some performance (and possibly other) problems, you might even call them bugs, with both map-index and the current implementation of cursors in the postmodern backend. Alex Mizrahi will reimplement things in a new way with the eye for performance, this work will probably be finished in some days. Best wishes, Henrik Hjelte -------------- next part -------------- An HTML attachment was scrubbed... URL: From killerstorm at newmail.ru Thu Nov 29 14:16:19 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 29 Nov 2007 16:16:19 +0200 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> Message-ID: AP> Am I missing something really basic here? actually it's quite strange situation that you have *many* employees with same name but you want just one (random one). i cannot imagine why one needs this in real world.. or you're saying that all have different names, but it still does consing? this could be a bug then.. From read at robertlread.net Thu Nov 29 15:06:17 2007 From: read at robertlread.net (Robert L. Read) Date: Thu, 29 Nov 2007 09:06:17 -0600 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases In-Reply-To: References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> Message-ID: <1196348777.3189.109.camel@penguin.yourdomain.com> On Thu, 2007-11-29 at 16:16 +0200, Alex Mizrahi wrote: > AP> Am I missing something really basic here? > > actually it's quite strange situation that you have *many* employees with > same name but you want just one (random one). i cannot imagine why one needs > this in real world.. > > or you're saying that all have different names, but it still does consing? > this could be a bug then.. > With respect to consing, it is important to point out that our serializer is very consing (for postmodern and CL-SQL backends.) This is because I used base64 to transform the byte-streams into character strings. Most relational databases (including Postgres) provide a way of storing byte sequences directly. However, this is not standardized and not portable. In fact, I spoke to Kevin Rosenberg, the author of CL-SQL, and he and CL-SQL don't have a good way to do it. However, since postmodern is Postgres specific, it could avoid this step, by using a back-end specific serializer. I suspect this would have a huge impact on performance, both by decreasing consing (minor) and by decreasing the amount of disc I/O that has to be done (major). (BDB doesn't have this problem, because it natively uses byte-sequences, not character-sequences.) Please see the code below, which demonstrates that pushing 1 million bytes through the serializer (without even going to the database) creates 8 million bytes of garbage in 0.433 seconds. (This is on a new, fast, 2 gigabyte 64-bit machine, against postmodern: asdf:operate 'asdf:load-op :elephant) (asdf:operate 'asdf:load-op :ele-clsql) (asdf:operate 'asdf:load-op :postmodern) (asdf:operate 'asdf:load-op :elephant-tests) (in-package "ELEPHANT-TESTS") (setq *default-spec* *testpm-spec*) (setq teststring "supercalifragiliciousexpialidocious") (setq testint 42) (setq totalseriazationload (* 1000 1000)) (setq n (ceiling (/ totalseriazationload (length teststring)))) (open-store *default-spec*) (time (dotimes (x n) (in-out-value teststring))) (close-store) ***** Results in: Evaluation took: 0.433 seconds of real time 0.172974 seconds of user run time 0.058991 seconds of system run time 0 calls to %EVAL 0 page faults and 8,731,728 bytes consed. NIL ELE-TESTS> I personally think making a back-end specific serializer to avoid the base64 encoding would make a significant performance difference. This is not much of an issue for me personally, since I keep everything cached in memory anyway. -- Robert L. Read, PhD http://konsenti.com From read at robertlread.net Thu Nov 29 15:09:50 2007 From: read at robertlread.net (Robert L. Read) Date: Thu, 29 Nov 2007 09:09:50 -0600 Subject: [elephant-devel] Severely poor performance in some obvious cases In-Reply-To: <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> Message-ID: <1196348990.3189.112.camel@penguin.yourdomain.com> I used CLSQL recording in the past when I have been debugging. I think that is the best way to do it; I wouldn't want to put such a think in Elephant, since it would be very backend-specific. On Wed, 2007-11-28 at 18:37 -0500, Ian Eslick wrote: > There is not SQL recording built into Elephant that I'm aware of, > however Henrik or Robert should weigh in on this as it may be easy > to > add. > From read at robertlread.net Thu Nov 29 18:43:39 2007 From: read at robertlread.net (Robert L. Read) Date: Thu, 29 Nov 2007 12:43:39 -0600 Subject: [elephant-devel] Severely poor performance in some obvious cases In-Reply-To: <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> Message-ID: <1196361819.3189.132.camel@penguin.yourdomain.com> On Wed, 2007-11-28 at 18:37 -0500, Ian Eslick wrote: > I'm surprised you are seeing this difference on Postmodern, unless > your input name is matching a large number of objects (i.e. s is > large). I had thought that the native PostgreSQL backend, > postmodern, > fixed the linear cost problem of CL-SQL indices and provides the > same > O(s) performance that BDB does, but I'm not certain (Henrik? > Robert?). It sounds like you are seeing a high linear cost. Postmodern does exactly the same thing that CL-SQL does in this respect; the code was copied directly from the CL-SQL code. From killerstorm at newmail.ru Thu Nov 29 18:40:41 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 29 Nov 2007 20:40:41 +0200 Subject: [elephant-devel] map-index Message-ID: hello i'm now trying to improve implementation of cursor in postmodern backend. it would be nice to optimize it for usage patterns of map-index, but map-index implementation frightens me: is there a reason for such complexity? for example, just to enumarate index with two values it does calls like this: (cursor-pfirst c) => 1 (cursor-pnext-dup c) => nil (cursor-pset c 1) (cursor-pnext-nodup c) => 2 (cursor-pnext-dup c) => nil (cursor-pset c 2) (cursor-pnext-nodup c) => nil while in my opinion it would be enough to do this: (cursor-pfirst c) => 1 (cursor-pnext c) => 2 (cursor-pnext c) => nil even if cursor is aware of such behaviour and caches cursor-pset, it's still a waste of CPU cycles to do 3 times more calls and demand cursor to do dup/nodup checks. next, if we have query by value, i think optimal sequence would be: (cursor-pset c value) => 1 (cursor-pnext-dup c) => 2 (cursor-pnext-dup c) => 3 (cursor-pnext-dup c) => nil ... that is, if cursor has enough logic to enumerate duplicates (all values for given key), why not use it? however when cursor-pnext-dup returns NIL, that means there's no more values with such key, elephant does once more (cursor-pset c value), and tries (cursor-pnext-nodup c), and then uses lisp-compare<= on returned value. this was actually a reason for error Henrik have reported. db-postmodern's ordering of keys differs from lisp-compare ordering (and in some cases it's just random). normally that should break only range queries. but with aforementioned (overly-general?) map-index logic it breaks even simple get-instances-by-value calls, which is not good. so, am i missing something, or really it would be better to refactor map-index? if it would be calling only what's needed, but won't do cursor-pset "for generality" i can skip cursor-pset optimization -- i don't think anybody in real code working with cursors will repeatadly call cursor-pset without a reason. with best regards, Alex 'killer_storm' Mizrahi. From killerstorm at newmail.ru Thu Nov 29 19:25:28 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Thu, 29 Nov 2007 21:25:28 +0200 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com><87hcj6n43l.fsf@memetrics.com><3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> <1196361819.3189.132.camel@penguin.yourdomain.com> Message-ID: ??>> I'm surprised you are seeing this difference on Postmodern, unless ??>> your input name is matching a large number of objects (i.e. s is ??>> large). I had thought that the native PostgreSQL backend, ??>> postmodern, ??>> fixed the linear cost problem of CL-SQL indices and provides the ??>> same ??>> O(s) performance that BDB does, but I'm not certain (Henrik? ??>> Robert?). It sounds like you are seeing a high linear cost. RLR> Postmodern does exactly the same thing that CL-SQL does in this RLR> respect; the code was copied directly from the CL-SQL code. from original Henrik's announcement: ---- The implementation is actually quite different from the clsql backend. ... The db-postmodern cursor is based on a sql cursor. Each btree has its own database table. ---- IE> SQL backend cannot support the ordering constraints of an index IE> without reading all the objects into memory and postmodern cannot support them too. however we do not really need to support ordering to do by-value queries. as for by-range queries, they should work in db-postmodern when keys are strings-only or integers-only, which is probably most useful case. we are going to improve it to support NILs among values. by the way, "ugly fix" Henrik mentions in original post is blocking range queries. for now we cannot even do even by-value queries because of "special way" map-index works, so "ugly fix" is needed. but we are going to provide optimized specialization for doing by-value queries, and fix stuff so by-range queries will work at least in some limited way. also in process of improving cursor implementation we are going to provide optional "compatible" mode that will allow degrading to CLSQL backend style (fetch everything and sort on Lisp side), so people who want to do funky range queries will be able to do so, but they will loose ability to work with arbitrarily large tables. it might be possible to implement lisp-compare in postgres somehow, but IMHO it doesn't worth effort to do so -- range queries make sense when you know type of data. From read at robertlread.net Thu Nov 29 21:04:32 2007 From: read at robertlread.net (Robert L. Read) Date: Thu, 29 Nov 2007 15:04:32 -0600 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases In-Reply-To: References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> <1196361819.3189.132.camel@penguin.yourdomain.com> Message-ID: <1196370273.3189.144.camel@penguin.yourdomain.com> On Thu, 2007-11-29 at 21:25 +0200, Alex Mizrahi wrote: > > RLR> Postmodern does exactly the same thing that CL-SQL does in this > RLR> respect; the code was copied directly from the CL-SQL code. > > from original Henrik's announcement: > > ---- > The implementation is actually quite different from > the clsql backend. > ... > The db-postmodern cursor is based on a sql cursor. Each btree has its > own database table. > ---- > > IE> SQL backend cannot support the ordering constraints of an index > IE> without reading all the objects into memory > > and postmodern cannot support them too. > however we do not really need to support ordering to do by-value > queries. > as for by-range queries, they should work in db-postmodern when keys > are > strings-only or integers-only, which is probably most useful case. we > are > going to improve it to support NILs among values. > I think this makes a lot of sense. From Alain.Picard at memetrics.com Thu Nov 29 23:25:34 2007 From: Alain.Picard at memetrics.com (Alain Picard) Date: Fri, 30 Nov 2007 10:25:34 +1100 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases In-Reply-To: (Alex Mizrahi's message of "Thu\, 29 Nov 2007 16\:16\:19 +0200") References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> Message-ID: <87k5o0prbl.fsf@memetrics.com> "Alex Mizrahi" writes: > AP> Am I missing something really basic here? > > actually it's quite strange situation that you have *many* employees > with same name but you want just one (random one). i cannot imagine > why one needs this in real world.. I didn't say they have the same name, at least I don't think I did. I have millions of employees, each with a unique name, and want to pull one back with an indexed, efficient search. > or you're saying that all have different names, but it still does > consing? Yes - that is what I am saying. cheers, --ap -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From Alain.Picard at memetrics.com Fri Nov 30 03:40:59 2007 From: Alain.Picard at memetrics.com (Alain Picard) Date: Fri, 30 Nov 2007 14:40:59 +1100 Subject: [elephant-devel] Severely poor performance in some obvious cases In-Reply-To: <50e8e4f60711290102k39102c40ucee3ec85f1a523f4@mail.gmail.com> (Henrik Hjelte's message of "Thu\, 29 Nov 2007 10\:02\:34 +0100") References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <3D15F975-54C8-4B3D-98D4-EA212C9E5ECD@csail.mit.edu> <50e8e4f60711290102k39102c40ucee3ec85f1a523f4@mail.gmail.com> Message-ID: <877ik0pfhw.fsf@memetrics.com> "Henrik Hjelte" writes: > We have recently discovered some performance (and possibly other) > problems, you might even call them bugs, with both map-index and the > current implementation of cursors in the postmodern backend. Alex > Mizrahi will reimplement things in a new way with the eye for > performance, this work will probably be finished in some days. Thank you. I'll be more than happy to test any reimplementation and comment on my findings. As each user tries to solve a slightly different problem, they tend to exercise the performance space in different directions, which I guess is probably a good thing. I really like the design of elephant, and I'm confident you guys will be able to iron out these issues as they get pointed out. --AP From eslick at csail.mit.edu Fri Nov 30 15:32:29 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 30 Nov 2007 10:32:29 -0500 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases In-Reply-To: <87k5o0prbl.fsf@memetrics.com> References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <87k5o0prbl.fsf@memetrics.com> Message-ID: <053E87CF-240F-4B05-AB98-9F22BE62CE3A@csail.mit.edu> Alain's solution to the get-instance-by-value is the correct one. It's an API issue, not a fundamental one. That is to say if you know you want only one of a potentially duplicate set from a slot index, get-instance-by-value should just use get-value on that index. I implemented that function a long time ago and didn't really think through it, it was just syntactic sugar for (car (get-instances-by-value)) which I ended up using alot. The map-index and cursor issues I will talk about in a follow-up e-mail. Ian On Nov 29, 2007, at 6:25 PM, Alain Picard wrote: > "Alex Mizrahi" writes: > >> AP> Am I missing something really basic here? >> >> actually it's quite strange situation that you have *many* employees >> with same name but you want just one (random one). i cannot imagine >> why one needs this in real world.. > > I didn't say they have the same name, at least I don't think I did. > I have millions of employees, each with a unique name, and > want to pull one back with an indexed, efficient search. > >> or you're saying that all have different names, but it still does >> consing? > > Yes - that is what I am saying. > > cheers, > --ap > > > -- > Please read about why Top Posting > is evil at: http://en.wikipedia.org/wiki/Top-posting > and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html > > Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Fri Nov 30 15:36:33 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 30 Nov 2007 10:36:33 -0500 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases In-Reply-To: <1196348777.3189.109.camel@penguin.yourdomain.com> References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <1196348777.3189.109.camel@penguin.yourdomain.com> Message-ID: <0AF89CF0-8AF1-4C50-AC91-1900DDFE5E42@csail.mit.edu> Is there any reason that we can't store the byte-stream data directly in postmodern? We already have an efficient, mostly non-consing byte- array serializer with the following format: [btree_id][data_type][data_format] If you used a new table for each btree, then you could strip the btree_id and pass the type + format to postgres. Integers are stored big-endian and strings in left to write lisp-code order so might just work without modification. However I'm speculating as I don't understand all the issues. As I understand it, CL-SQL has the problem that the byte storage methods for different SQL engines are different enough to make a common API difficult to implement. Ian On Nov 29, 2007, at 10:06 AM, Robert L. Read wrote: > On Thu, 2007-11-29 at 16:16 +0200, Alex Mizrahi wrote: >> AP> Am I missing something really basic here? >> >> actually it's quite strange situation that you have *many* >> employees with >> same name but you want just one (random one). i cannot imagine why >> one needs >> this in real world.. >> >> or you're saying that all have different names, but it still does >> consing? >> this could be a bug then.. >> > > With respect to consing, it is important to point out that our > serializer is very consing (for postmodern and CL-SQL backends.) This > is because I used base64 to transform the byte-streams into character > strings. > > Most relational databases (including Postgres) provide a way of > storing > byte sequences directly. However, this is not standardized and not > portable. In fact, I spoke to Kevin Rosenberg, the author of CL-SQL, > and he and CL-SQL don't have a good way to do it. > > However, since postmodern is Postgres specific, it could avoid this > step, by using a back-end specific serializer. I suspect this would > have a huge impact on performance, both by decreasing consing (minor) > and by decreasing the amount of disc I/O that has to be done (major). > > (BDB doesn't have this problem, because it natively uses byte- > sequences, > not character-sequences.) > > Please see the code below, which demonstrates that pushing 1 million > bytes through the serializer (without even going to the database) > creates 8 million bytes of garbage in 0.433 seconds. (This is on a > new, > fast, 2 gigabyte 64-bit machine, against postmodern: > > asdf:operate 'asdf:load-op :elephant) > (asdf:operate 'asdf:load-op :ele-clsql) > (asdf:operate 'asdf:load-op :postmodern) > > (asdf:operate 'asdf:load-op :elephant-tests) > (in-package "ELEPHANT-TESTS") > > (setq *default-spec* *testpm-spec*) > > (setq teststring "supercalifragiliciousexpialidocious") > (setq testint 42) > > (setq totalseriazationload (* 1000 1000)) > > (setq n (ceiling (/ totalseriazationload (length teststring)))) > > (open-store *default-spec*) > > (time > (dotimes (x n) > (in-out-value teststring))) > > (close-store) > > ***** > Results in: > Evaluation took: > 0.433 seconds of real time > 0.172974 seconds of user run time > 0.058991 seconds of system run time > 0 calls to %EVAL > 0 page faults and > 8,731,728 bytes consed. > NIL > ELE-TESTS> > > I personally think making a back-end specific serializer to avoid the > base64 encoding would make a significant performance difference. This > is not much of an issue for me personally, since I keep everything > cached in memory anyway. > > > -- > Robert L. Read, PhD > http://konsenti.com > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Fri Nov 30 15:45:37 2007 From: read at robertlread.net (Robert L. Read) Date: Fri, 30 Nov 2007 09:45:37 -0600 Subject: [elephant-devel] Re: Severely poor performance in some obvious cases In-Reply-To: <0AF89CF0-8AF1-4C50-AC91-1900DDFE5E42@csail.mit.edu> References: <50e8e4f60711230700qa33ad58s1419c20bc7afd146@mail.gmail.com> <87hcj6n43l.fsf@memetrics.com> <1196348777.3189.109.camel@penguin.yourdomain.com> <0AF89CF0-8AF1-4C50-AC91-1900DDFE5E42@csail.mit.edu> Message-ID: <1196437537.3189.216.camel@penguin.yourdomain.com> On Fri, 2007-11-30 at 10:36 -0500, Ian Eslick wrote: > Is there any reason that we can't store the byte-stream data directly > in postmodern? We already have an efficient, mostly non-consing byte- > array serializer with the following format: > > [btree_id][data_type][data_format] > > If you used a new table for each btree, then you could strip the > btree_id and pass the type + format to postgres. Integers are stored > big-endian and strings in left to write lisp-code order so might just > work without modification. I think I was wrong about the base64 stuff; it is used by CL-SQL but not by postmodern. Sorry for the confusion. > > However I'm speculating as I don't understand all the issues. As I > understand it, CL-SQL has the problem that the byte storage methods > for different SQL engines are different enough to make a common API > difficult to implement. Yes, that is correct, but postmodern has done better and eliminated this problem. Nonetheless, it seems by experiment that our serializer is very, very consing. > > Ian > > > On Nov 29, 2007, at 10:06 AM, Robert L. Read wrote: > > > On Thu, 2007-11-29 at 16:16 +0200, Alex Mizrahi wrote: > >> AP> Am I missing something really basic here? > >> > >> actually it's quite strange situation that you have *many* > >> employees with > >> same name but you want just one (random one). i cannot imagine why > >> one needs > >> this in real world.. > >> > >> or you're saying that all have different names, but it still does > >> consing? > >> this could be a bug then.. > >> > > > > With respect to consing, it is important to point out that our > > serializer is very consing (for postmodern and CL-SQL backends.) This > > is because I used base64 to transform the byte-streams into character > > strings. > > > > Most relational databases (including Postgres) provide a way of > > storing > > byte sequences directly. However, this is not standardized and not > > portable. In fact, I spoke to Kevin Rosenberg, the author of CL-SQL, > > and he and CL-SQL don't have a good way to do it. > > > > However, since postmodern is Postgres specific, it could avoid this > > step, by using a back-end specific serializer. I suspect this would > > have a huge impact on performance, both by decreasing consing (minor) > > and by decreasing the amount of disc I/O that has to be done (major). > > > > (BDB doesn't have this problem, because it natively uses byte- > > sequences, > > not character-sequences.) > > > > Please see the code below, which demonstrates that pushing 1 million > > bytes through the serializer (without even going to the database) > > creates 8 million bytes of garbage in 0.433 seconds. (This is on a > > new, > > fast, 2 gigabyte 64-bit machine, against postmodern: > > > > asdf:operate 'asdf:load-op :elephant) > > (asdf:operate 'asdf:load-op :ele-clsql) > > (asdf:operate 'asdf:load-op :postmodern) > > > > (asdf:operate 'asdf:load-op :elephant-tests) > > (in-package "ELEPHANT-TESTS") > > > > (setq *default-spec* *testpm-spec*) > > > > (setq teststring "supercalifragiliciousexpialidocious") > > (setq testint 42) > > > > (setq totalseriazationload (* 1000 1000)) > > > > (setq n (ceiling (/ totalseriazationload (length teststring)))) > > > > (open-store *default-spec*) > > > > (time > > (dotimes (x n) > > (in-out-value teststring))) > > > > (close-store) > > > > ***** > > Results in: > > Evaluation took: > > 0.433 seconds of real time > > 0.172974 seconds of user run time > > 0.058991 seconds of system run time > > 0 calls to %EVAL > > 0 page faults and > > 8,731,728 bytes consed. > > NIL > > ELE-TESTS> > > > > I personally think making a back-end specific serializer to avoid the > > base64 encoding would make a significant performance difference. This > > is not much of an issue for me personally, since I keep everything > > cached in memory anyway. > > > > > > -- > > Robert L. Read, PhD > > http://konsenti.com > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Fri Nov 30 15:54:20 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 30 Nov 2007 10:54:20 -0500 Subject: [elephant-devel] map-index In-Reply-To: References: Message-ID: <07748390-89A3-4F8F-A4E0-34767A69C5C6@csail.mit.edu> Yeah, map-index is complex but that's due to problems with cursors on secondary btrees in Elephant. I'd like to rewrite it but didn't find a good strategy before. See my notes below. I have one idea I'll look into for the standard case of walking a simple range. map-index is generic, you are welcome to override it with a postmodern specific version rather than stressing about making the cursor APIs work such that they make map-index work. On Nov 29, 2007, at 1:40 PM, Alex Mizrahi wrote: > hello > > i'm now trying to improve implementation of cursor in postmodern > backend. > it would be nice to optimize it for usage patterns of map-index, but > map-index implementation frightens me: is there a reason for such > complexity? > > for example, just to enumarate index with two values it does calls > like > this: > > (cursor-pfirst c) => 1 > (cursor-pnext-dup c) => nil > (cursor-pset c 1) > (cursor-pnext-nodup c) => 2 > (cursor-pnext-dup c) => nil > (cursor-pset c 2) > (cursor-pnext-nodup c) => nil > > while in my opinion it would be enough to do this: > > (cursor-pfirst c) => 1 > (cursor-pnext c) => 2 > (cursor-pnext c) => nil What if there are more than two values = 2? This will only get the first one. In this case, you need to get all duplicates for each value. The pset is used because pnext-dup = nil causes the cursor to have an invalid location (Elephant design problem) so we use pset to get back into the current duplicate set so we can do a pnext-no-dup on a valid cursor. If your pnext-dup can return nil and still point at the last element of the duplicate set, then you could remove that call in your version of map-index. > even if cursor is aware of such behaviour and caches cursor-pset, > it's still > a waste of CPU cycles to do 3 times more calls and demand cursor to do > dup/nodup checks. Althoughthese calls are probably not a big deal because the pages are cached in memory and page fetch time swamps a few function calls so it's in the noise for overall performance. For really large DBs there may be issues keeping all the higher pages in memory to support pset (since pnext only fetches adjacent pages). I'll think about this a bit... I think this is basically what happens since you don't need to do a pset when pnext-dup returns nil. > next, if we have query by value, i think optimal sequence would be: > > (cursor-pset c value) => 1 > (cursor-pnext-dup c) => 2 > (cursor-pnext-dup c) => 3 > (cursor-pnext-dup c) => nil > ... > > that is, if cursor has enough logic to enumerate duplicates (all > values for > given key), why not use it? > however when cursor-pnext-dup returns NIL, that means there's no > more values > with such key, elephant does once more (cursor-pset c value), and > tries > (cursor-pnext-nodup c), and then uses lisp-compare<= on returned > value. > > this was actually a reason for error Henrik have reported. > db-postmodern's ordering of keys differs from lisp-compare ordering > (and in > some cases it's just random). normally that should break only range > queries. > but with aforementioned (overly-general?) map-index logic it breaks > even > simple get-instances-by-value calls, which is not good. > > so, am i missing something, or really it would be better to refactor > map-index? > > if it would be calling only what's needed, but won't do cursor-pset > "for > generality" i can skip cursor-pset optimization -- i don't think > anybody in > real code working with cursors will repeatadly call cursor-pset > without a > reason. > > with best regards, Alex 'killer_storm' Mizrahi. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Fri Nov 30 17:24:17 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Fri, 30 Nov 2007 19:24:17 +0200 Subject: [elephant-devel] Re: map-index References: <07748390-89A3-4F8F-A4E0-34767A69C5C6@csail.mit.edu> Message-ID: ??>> while in my opinion it would be enough to do this: ??>> ??>> (cursor-pfirst c) => 1 ??>> (cursor-pnext c) => 2 ??>> (cursor-pnext c) => nil IE> What if there are more than two values = 2? This will only get the IE> first one. so you say that cursor-pnext is same as cursor-pnext-nodup? i thought that it just iterates sequence regardless whether there are duplicates or not -- just what we want. the BDB documentation says this: ---- Otherwise, the cursor is moved to the next key/data pair of the database, and that pair is returned. In the presence of duplicate key values, the value of the key may not change. ---- looks like it means it works fine with duplicate keys, but wording is vague.. From eslick at csail.mit.edu Fri Nov 30 19:34:10 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 30 Nov 2007 14:34:10 -0500 Subject: [elephant-devel] Re: map-index In-Reply-To: References: <07748390-89A3-4F8F-A4E0-34767A69C5C6@csail.mit.edu> Message-ID: <27182B3C-AC32-4274-AA24-EFC9B3D307C5@csail.mit.edu> Well, a quick experiment proves your point. I'm not sure why I didn't take advantage of that earlier. There were a bunch of small surprising issues that I recall making the code unpleasant, but I can't recall the exact reasons. Given all the tests we have covering this now, I think it's safe to create a parallel version that should be cleaner and see if you/we can improve on this otherwise messy creation. Ian On Nov 30, 2007, at 12:24 PM, Alex Mizrahi wrote: > ??>> while in my opinion it would be enough to do this: > ??>> > ??>> (cursor-pfirst c) => 1 > ??>> (cursor-pnext c) => 2 > ??>> (cursor-pnext c) => nil > > IE> What if there are more than two values = 2? This will only get > the > IE> first one. > > so you say that cursor-pnext is same as cursor-pnext-nodup? > i thought that it just iterates sequence regardless whether there are > duplicates or not -- just what we want. > > the BDB documentation says this: > ---- > Otherwise, the cursor is moved to the next key/data pair of the > database, > and that pair is returned. In the presence of duplicate key values, > the > value of the key may not change. > ---- > > looks like it means it works fine with duplicate keys, but wording is > vague.. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Fri Nov 30 22:36:16 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 30 Nov 2007 17:36:16 -0500 Subject: [elephant-devel] Re: map-index In-Reply-To: References: <07748390-89A3-4F8F-A4E0-34767A69C5C6@csail.mit.edu> Message-ID: <3B8F410C-6A33-4B88-A893-B92D5774309C@csail.mit.edu> I just pushed a patch to the main repository that I hope made the map- index a little easier to read (if you ignore the variable capture stuff in the macros) and also makes it more efficient along the lines of what Alex suggests. This passes all tests under BDB. Please check it out! Ian On Nov 30, 2007, at 12:24 PM, Alex Mizrahi wrote: > ??>> while in my opinion it would be enough to do this: > ??>> > ??>> (cursor-pfirst c) => 1 > ??>> (cursor-pnext c) => 2 > ??>> (cursor-pnext c) => nil > > IE> What if there are more than two values = 2? This will only get > the > IE> first one. > > so you say that cursor-pnext is same as cursor-pnext-nodup? > i thought that it just iterates sequence regardless whether there are > duplicates or not -- just what we want. > > the BDB documentation says this: > ---- > Otherwise, the cursor is moved to the next key/data pair of the > database, > and that pair is returned. In the presence of duplicate key values, > the > value of the key may not change. > ---- > > looks like it means it works fine with duplicate keys, but wording is > vague.. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel