From eslick at media.mit.edu Mon Feb 2 20:29:50 2009 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 2 Feb 2009 15:29:50 -0500 Subject: [elephant-devel] Lisp-only data store (prototype) Message-ID: <9117CD2A-1CBA-4CE0-A098-8C6E3B50CBE2@media.mit.edu> Hi All, I was inspired the other day to come up with a minimal, quick-and- dirty, all-lisp data store for Elephant based on cl-prevalence. The problem with cl-prevalence is that every access to a persistent slot has to be explicitly transactionally protected so it can be recovered. This is onerous and violates the abstractions provided by the rest of the elephant data stores. The general idea is to hide the cl-prevalence transaction model inside the existing meta-object protocol. Instead of the usual hash table, we use trees from cl-collections as our replacement for btrees and indexes - this gives us reasonably efficient successor/predecessor ops and makes it easier to implement the mapping and cursor APIs. It's not going to be nearly as fast as a full prevalence solution, but it will be faster than BDB and should be much easier to install and work with on a single-image basis. The new db-clp store requires a patches to cl-prevalence and cl-containers. I'll see about getting those put into the mainstream, but can send them to anyone interesting in hacking on this in the meantime. I checked a prototype of this into elephant-1.0 today; it does not interfere with existing functionality at all and I'm not sure yet whether it will be part of 1.0. You can open a new store, subject to the following caveats, by: In elephant root: ln -s src/contrib/eslick/db-clp/ele-clp.asd ele- clp.asd (open-store '(:CLP "/home/me/db/")) where /home/me/db/ is a fresh directory. 85%+ of the tests pass and a ton of stuff does work: persistent classes, btrees, dup-btrees, indexed-btrees, mapping (mostly), and class indexing (mostly). However, there are still some very serious holes. The ones I'm aware of are: - You can only create, but not re-open a store (this is due to a bootstrapping problem in recreating persistent instances when loading a snapshot) - No cursors are supported (all those tests fail), but should be easy as there is a good match between the RB tree and the btree abstractions but you will need to add a couple of functions to cl- containers. - On-line recovery has not been tested. - I'm unsure of how cl-prevalence guarantees global transaction serializability - I don't see locks anywhere in the code. - I faked out a bunch of serializer tests by including the default serializer, because they depend on buffer streams which the xml serializer doesn't support; the tests should be fixed to be more general and not rely on buffer streams. - Some issues in schema evolution tests Performance issues: - Reads need to be serialized so are currently considered transactions for simplicity, but they write a transaction log - they should be rewritten to only use the serialization mechanism but not write a log and the grabbing of a lock during serialization should be done once per with-transaction call. - with-transaction is a no-op, a significant performance enhancement would be to bundle a set of primitive operations such as tx-write- slot, tx-remove-kv and write a single prevalence log entry for them. This would avoid a disk write per primop. - I started using splay trees, retreated to RB trees due to bugs and finally retreated to binary-search-trees due to more bugs. Moving to a more efficient, balanced tree data structure will improve the performance of all operations. However, several tests use linear insertion, reducing the asymptotic behavior of the tree to that of a list - it really grinds the tests to a halt. Obvious next steps, in order of increasing difficulty: - Implement cursor API - Fix red-black or splay tree implementation in cl-containers - Figure out how to load from a snapshot - Use with-transaction to improve performance - Read transaction optimizations Regards, Ian From elliottslaughter at gmail.com Mon Feb 2 20:58:58 2009 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Mon, 2 Feb 2009 12:58:58 -0800 Subject: [elephant-devel] Lisp-only data store (prototype) In-Reply-To: <9117CD2A-1CBA-4CE0-A098-8C6E3B50CBE2@media.mit.edu> References: <9117CD2A-1CBA-4CE0-A098-8C6E3B50CBE2@media.mit.edu> Message-ID: <42c0ab790902021258m50f33905j9714ae7da343717@mail.gmail.com> Hi, Thanks for working on this. I am particularly interested in the potential of this backend because I need (a) speed and (b) the ability to build a binary which will run on end-user machines without requiring them to install a separate library. If you send me the necessary patches (or get them integrated into their respective libraries), I would be happy to help you test this code on my application. I look forward to hearing from you. On Mon, Feb 2, 2009 at 12:29 PM, Ian Eslick wrote: > Hi All, > > I was inspired the other day to come up with a minimal, quick-and- > dirty, all-lisp data store for Elephant based on cl-prevalence. The > problem with cl-prevalence is that every access to a persistent slot > has to be explicitly transactionally protected so it can be > recovered. This is onerous and violates the abstractions provided by > the rest of the elephant data stores. > > The general idea is to hide the cl-prevalence transaction model inside > the existing meta-object protocol. Instead of the usual hash table, > we use trees from cl-collections as our replacement for btrees and > indexes - this gives us reasonably efficient successor/predecessor ops > and makes it easier to implement the mapping and cursor APIs. > > It's not going to be nearly as fast as a full prevalence solution, but > it will be faster than BDB and should be much easier to install and > work with on a single-image basis. The new db-clp store requires a > patches to cl-prevalence and cl-containers. I'll see about getting > those put into the mainstream, but can send them to anyone interesting > in hacking on this in the meantime. > > I checked a prototype of this into elephant-1.0 today; it does not > interfere with existing functionality at all and I'm not sure yet > whether it will be part of 1.0. You can open a new store, subject to > the following caveats, by: > > In elephant root: ln -s src/contrib/eslick/db-clp/ele-clp.asd ele- > clp.asd > > (open-store '(:CLP "/home/me/db/")) > > where /home/me/db/ is a fresh directory. > > 85%+ of the tests pass and a ton of stuff does work: persistent > classes, btrees, dup-btrees, indexed-btrees, mapping (mostly), and > class indexing (mostly). > > > However, there are still some very serious holes. > The ones I'm aware of are: > > - You can only create, but not re-open a store > (this is due to a bootstrapping problem in recreating persistent > instances > when loading a snapshot) > > - No cursors are supported (all those tests fail), but should be easy > as there is a good match between the RB tree and the btree > abstractions but you will need to add a couple of functions to cl- > containers. > > - On-line recovery has not been tested. > > - I'm unsure of how cl-prevalence guarantees global transaction > serializability - I don't see locks anywhere in the code. > > - I faked out a bunch of serializer tests by including the default > serializer, > because they depend on buffer streams which the xml serializer > doesn't support; > the tests should be fixed to be more general and not rely on buffer > streams. > > - Some issues in schema evolution tests > > Performance issues: > > - Reads need to be serialized so are currently considered transactions > for simplicity, but they write a transaction log - they should be > rewritten to only use the serialization mechanism but not write a log > and the grabbing of a lock during serialization should be done once > per with-transaction call. > > - with-transaction is a no-op, a significant performance enhancement > would be to bundle a set of primitive operations such as tx-write- > slot, tx-remove-kv and write a single prevalence log entry for them. > This would avoid a disk write per primop. > > - I started using splay trees, retreated to RB trees due to bugs and > finally retreated to binary-search-trees due to more bugs. Moving to > a more efficient, balanced tree data structure will improve the > performance of all operations. However, several tests use linear > insertion, reducing the asymptotic behavior of the tree to that of a > list - it really grinds the tests to a halt. > > Obvious next steps, in order of increasing difficulty: > - Implement cursor API > - Fix red-black or splay tree implementation in cl-containers > - Figure out how to load from a snapshot > - Use with-transaction to improve performance > - Read transaction optimizations > > Regards, > Ian > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From elliottslaughter at gmail.com Mon Feb 2 23:45:48 2009 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Mon, 2 Feb 2009 15:45:48 -0800 Subject: [elephant-devel] Windows/SBCL bug (AND FIX) when compiling DLLs (WAS: partial fix) In-Reply-To: <42c0ab790901080114l4821f60fo8051b7ffca6d55b9@mail.gmail.com> References: <42c0ab790901080114l4821f60fo8051b7ffca6d55b9@mail.gmail.com> Message-ID: <42c0ab790902021545y7270723by3b4a9dd61bb802c0@mail.gmail.com> Just wondering if there is any chance this patch will be reviewed and/or integrated soon? Thanks. On Thu, Jan 8, 2009 at 1:14 AM, Elliott Slaughter < elliottslaughter at gmail.com> wrote: > Hi, > > On Mon, Dec 15, 2008 at 11:03 PM, Elliott Slaughter < > elliottslaughter at gmail.com> wrote: >> >> I just encountered a peculiarity with the perform method for >> elephant-c-source files, which causes a clean build of elephant-unstable on >> Windows/SBCL to fail. >> > > Attached is a patch which fixes this problem in a better way, and safely > deals with different drive letters and paths with spaces. > > Enjoy :-) > > -- > Elliott Slaughter > > "Any road followed precisely to its end leads precisely nowhere." - Frank > Herbert > -- Elliott Slaughter "Any road followed precisely to its end leads precisely nowhere." - Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: better-support-for-__prebuilt_libraries-nil_-under-win32_sbcl_.dpatch Type: application/octet-stream Size: 22321 bytes Desc: not available URL: From larry at theclapp.org Tue Feb 3 01:41:38 2009 From: larry at theclapp.org (Larry Clapp) Date: Mon, 2 Feb 2009 20:41:38 -0500 Subject: [elephant-devel] Elephant won't compile under Lispworks for Linux 5.1 Message-ID: <20090203014137.GA1720@cupid.theclapp.org> Is there anything I can do to improve on the below error report? I'd like to use elephant but it doesn't compile on Lispworks for Linux 5.1. Am I missing something? Any help appreciated. -- Larry ----- Forwarded message from Larry Clapp ----- > From: Larry Clapp > Date: Sun, 25 Jan 2009 16:30:36 -0500 > To: elephant-devel at common-lisp.net > Subject: Elephant won't compile under Lispworks for Linux 5.1 > > This appears to be similar to the error reported back in April 08 > (http://common-lisp.net/pipermail/elephant-devel/2008-April/003925.html). > > Search below for "++++ Error". > > -- Larry > > > CL-USER 1 > (asdf :elephant) > ; loading system definition from /home/lmc/lisp/systems/elephant.asd into [snip] > ; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/metaclasses.ufasl > ;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/elephant/classes.lisp ... > ;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0 > ;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3 > ;;; Source level debugging is on > ;;; Source file recording is on > ;;; Cross referencing is on > ; (TOP-LEVEL-FORM 0) > ; (TOP-LEVEL-FORM 1) > ; (DEFVAR ELEPHANT::*DEBUG-SI*) > ; (METHOD INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) > ; ELEPHANT::INITIAL-PERSISTENT-SETUP > ; ELEPHANT::REGISTER-NEW-INSTANCE > ; ELEPHANT::CHECK-VALID-STORE-CONTROLLER > ; (DEFCLASS ELEPHANT:PERSISTENT-COLLECTION) > ; (DEFCLASS ELEPHANT:PERSISTENT-OBJECT) > > **++++ Error in (DEFCLASS ELEPHANT::CACHEABLE-PERSISTENT-OBJECT): > Invalid allocation type INSTANCE for slot-definition-allocation > ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-METACLASS T)) [snip] ----- End forwarded message ----- From binghe.lisp at gmail.com Tue Feb 3 02:50:17 2009 From: binghe.lisp at gmail.com (Chun Tian (binghe)) Date: Tue, 3 Feb 2009 10:50:17 +0800 Subject: [elephant-devel] Elephant won't compile under Lispworks for Linux 5.1 In-Reply-To: <20090203014137.GA1720@cupid.theclapp.org> References: <20090203014137.GA1720@cupid.theclapp.org> Message-ID: <1D3B04DB-AB49-4E96-B771-7DC3E9EBF74B@gmail.com> Hi, Larry I just checked, Elephant 1.0 on my LWL64 5.0.2 can be compiled and loaded correctly. LWL64 5.1.2 failed on classes.lisp which you met. In src/elephant/metaclasses.lisp, I found two definitions which mentioned about LispWorks versions: line 396: #+(or :lispworks3 :lispworks4 (and :lispworks5 :lispworks5.0)) (defmethod (setf slot-definition-allocation) (allocation (slot-def persistent-slot-definition)) (unless (eq allocation :database) (error "Invalid allocation type ~A for slot-definition- allocation" allocation)) allocation) line 587: #+lispworks (defmethod (setf slot-definition-allocation) (allocation (slot-def persistent-slot-definition)) (unless (eq allocation :database) (error "Invalid allocation type ~A for slot-definition- allocation" allocation)) allocation) The error output you saw, is reported by the second method definition above, I believe. I don't know much about Elephant, but I think above two method should both be disabled in LispWorks 5.1, which confirms "slot-definition changes for AMOP compatibility" in LW5.1: http://www.lispworks.com/documentation/lw51/RNIG/html/readme-364.htm#pgfId-918212 -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant-lw51-1.diff Type: application/octet-stream Size: 709 bytes Desc: not available URL: -------------- next part -------------- With my trivial patch, I go further: ;;; Compiling file /mnt/hgfs/Lisp/packages/elephant-1.0/src/elephant/ classes.lisp ... ;;; Safety = 3, Speed = 1, Space = 1, Float = 1, Interruptible = 0 ;;; Compilation speed = 1, Debug = 2, Fixnum safety = 3 ;;; Source level debugging is on ;;; Source file recording is on ;;; Cross referencing is on ; (TOP-LEVEL-FORM 0) ; (TOP-LEVEL-FORM 1) ; (DEFVAR ELEPHANT::*DEBUG-SI*) ; (METHOD INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) ; ELEPHANT::INITIAL-PERSISTENT-SETUP ; ELEPHANT::REGISTER-NEW-INSTANCE ; ELEPHANT::CHECK-VALID-STORE-CONTROLLER ; (DEFCLASS ELEPHANT:PERSISTENT-COLLECTION) ; (DEFCLASS ELEPHANT:PERSISTENT-OBJECT) ; (DEFCLASS ELEPHANT::CACHEABLE-PERSISTENT-OBJECT) ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-METACLASS T)) ; ELEPHANT::ENSURE-CLASS-INHERITS-FROM ; ELEPHANT::SUPERCLASS-MEMBER-P ; ELEPHANT::HAS-CACHED-SLOT-SPECIFICATION ; (METHOD CLOS:FINALIZE-INHERITANCE :AFTER (ELEPHANT:PERSISTENT-METACLASS)) ; ELEPHANT::COMPUTE-DERIVED-INDEX-TRIGGERS ; (METHOD REINITIALIZE-INSTANCE :AROUND (ELEPHANT:PERSISTENT-METACLASS)) ; ELEPHANT::BIND-SLOT-DEFS ; ELEPHANT::COMPUTE-BINDINGS ; (METHOD INITIALIZE-INSTANCE :AROUND (ELEPHANT:PERSISTENT-OBJECT)) ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-OBJECT T)) **++++ Error in (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT::CACHEABLE-PERSISTENT-OBJECT T)): Error during finalization of class #: Dynamic-Slot-Boundp-Using-Slotd is not defined for slot PCHECKED-OUT in # ; ELEPHANT::INITIALIZE-PERSISTENT-SLOTS ; ELEPHANT::INITIALIZE-SET-SLOTS ; ELEPHANT::INITIALIZE-FROM-INITARG ; ELEPHANT::GET-INIT-SLOTNAMES ; (SUBFUNCTION (DEFCLASS ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA)) ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) ; (DEFINE-CONDITION ELEPHANT:DROPPING-PERSISTENT-SLOT-DATA) ; ELEPHANT::WARN-ABOUT-DROPPED-SLOTS ; (METHOD ELEPHANT::RECREATE-INSTANCE-USING-CLASS (T)) ; (DEFGENERIC ELEPHANT::RECREATE-INSTANCE) ; (METHOD ELEPHANT::RECREATE-INSTANCE (ELEPHANT:PERSISTENT-OBJECT)) ; (METHOD ELEPHANT::RECREATE-INSTANCE (ELEPHANT:PERSISTENT-COLLECTION)) ; (METHOD UPDATE-INSTANCE-FOR-REDEFINED-CLASS :AROUND (ELEPHANT:PERSISTENT-OBJECT T T T)) ; (METHOD CHANGE-CLASS :BEFORE (ELEPHANT:PERSISTENT STANDARD-CLASS)) ; (METHOD UPDATE-INSTANCE-FOR-DIFFERENT-CLASS :AFTER (ELEPHANT:PERSISTENT-OBJECT ELEPHANT:PERSISTENT-OBJECT)) ; (METHOD CHANGE-CLASS :BEFORE (STANDARD-OBJECT ELEPHANT:PERSISTENT-METACLASS)) ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION)) ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION)) ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION)) ; (METHOD CLOS:SLOT-BOUNDP-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT SYMBOL)) ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT ELEPHANT::PERSISTENT-SLOT-DEFINITION)) ; ELEPHANT::VALID-PERSISTENT-REFERENCE-P ; (SUBFUNCTION (DEFCLASS ELEPHANT:CROSS-REFERENCE-ERROR) (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR)) ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) ; (DEFINE-CONDITION ELEPHANT:CROSS-REFERENCE-ERROR) ; ELEPHANT::SIGNAL-CROSS-REFERENCE-ERROR ; (METHOD CLOS:SLOT-VALUE-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT T)) ; (METHOD (SETF CLOS:SLOT-VALUE-USING-CLASS) (T ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT T)) ; (METHOD CLOS:SLOT-MAKUNBOUND-USING-CLASS (ELEPHANT:PERSISTENT-METACLASS ELEPHANT:PERSISTENT-OBJECT T)) ; (TOP-LEVEL-FORM 0) ; *** 1 error detected, no fasl file produced. I think above compile error should be fixed by a simple patch, but I don't know how to fix it. Hope other guys better than me can help us go further. There're some MOP differences between LispWorks 5.1 and early versions which mentioned in LW5.1 release notes: http://www.lispworks.com/documentation/lw51/RNIG/html/readme-362.htm#pgfId-917070 Regards, Chun Tian (binghe) On 2009-2-3, at 09:41, Larry Clapp wrote: > Is there anything I can do to improve on the below error report? I'd > like to use elephant but it doesn't compile on Lispworks for Linux > 5.1. Am I missing something? Any help appreciated. > > -- Larry > > ----- Forwarded message from Larry Clapp ----- > >> From: Larry Clapp >> Date: Sun, 25 Jan 2009 16:30:36 -0500 >> To: elephant-devel at common-lisp.net >> Subject: Elephant won't compile under Lispworks for Linux 5.1 >> >> This appears to be similar to the error reported back in April 08 >> (http://common-lisp.net/pipermail/elephant-devel/2008-April/003925.html >> ). >> >> Search below for "++++ Error". >> >> -- Larry >> >> >> CL-USER 1 > (asdf :elephant) >> ; loading system definition from /home/lmc/lisp/systems/ >> elephant.asd into > [snip] >> ; Loading fasl file /home/lmc/lisp/elephant/elephant-1.0/src/ >> elephant/metaclasses.ufasl >> ;;; Compiling file /home/lmc/lisp/elephant/elephant-1.0/src/ >> elephant/classes.lisp ... >> ;;; Safety = 1, Speed = 3, Space = 1, Float = 1, Interruptible = 0 >> ;;; Compilation speed = 0, Debug = 3, Fixnum safety = 3 >> ;;; Source level debugging is on >> ;;; Source file recording is on >> ;;; Cross referencing is on >> ; (TOP-LEVEL-FORM 0) >> ; (TOP-LEVEL-FORM 1) >> ; (DEFVAR ELEPHANT::*DEBUG-SI*) >> ; (METHOD INITIALIZE-INSTANCE :BEFORE (ELEPHANT:PERSISTENT)) >> ; ELEPHANT::INITIAL-PERSISTENT-SETUP >> ; ELEPHANT::REGISTER-NEW-INSTANCE >> ; ELEPHANT::CHECK-VALID-STORE-CONTROLLER >> ; (DEFCLASS ELEPHANT:PERSISTENT-COLLECTION) >> ; (DEFCLASS ELEPHANT:PERSISTENT-OBJECT) >> >> **++++ Error in (DEFCLASS ELEPHANT::CACHEABLE-PERSISTENT-OBJECT): >> Invalid allocation type INSTANCE for slot-definition-allocation >> ; (METHOD SHARED-INITIALIZE :AROUND (ELEPHANT:PERSISTENT-METACLASS >> T)) > [snip] > > ----- End forwarded message ----- > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -- Chun Tian (binghe) NetEase.com, Inc. P. R. China From eslick at media.mit.edu Tue Feb 3 04:04:22 2009 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 2 Feb 2009 23:04:22 -0500 Subject: [elephant-devel] Windows/SBCL bug (AND FIX) when compiling DLLs (WAS: partial fix) In-Reply-To: <42c0ab790902021545y7270723by3b4a9dd61bb802c0@mail.gmail.com> References: <42c0ab790901080114l4821f60fo8051b7ffca6d55b9@mail.gmail.com> <42c0ab790902021545y7270723by3b4a9dd61bb802c0@mail.gmail.com> Message-ID: Hi Elliott, My apologies, this dropped off my radar. I'll review and promote tomorrow. If anyone else has patches that haven't been committed, please re-send them. I'll put in a fix for the lispworks bug as well. Ian On Feb 2, 2009, at 6:45 PM, Elliott Slaughter wrote: > Just wondering if there is any chance this patch will be reviewed > and/or integrated soon? > > Thanks. > > On Thu, Jan 8, 2009 at 1:14 AM, Elliott Slaughter > wrote: > Hi, > > On Mon, Dec 15, 2008 at 11:03 PM, Elliott Slaughter > wrote: > I just encountered a peculiarity with the perform method for > elephant-c-source files, which causes a clean build of elephant- > unstable on Windows/SBCL to fail. > > Attached is a patch which fixes this problem in a better way, and > safely deals with different drive letters and paths with spaces. > > Enjoy :-) > > -- > Elliott Slaughter > > "Any road followed precisely to its end leads precisely nowhere." - > Frank Herbert > > > > -- > Elliott Slaughter > > "Any road followed precisely to its end leads precisely nowhere." - > Frank Herbert > win32_sbcl_.dpatch>_______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Tue Feb 3 10:08:40 2009 From: henrik at evahjelte.com (Henrik Hjelte) Date: Tue, 3 Feb 2009 11:08:40 +0100 Subject: [elephant-devel] Lisp-only data store (prototype) In-Reply-To: <9117CD2A-1CBA-4CE0-A098-8C6E3B50CBE2@media.mit.edu> References: <9117CD2A-1CBA-4CE0-A098-8C6E3B50CBE2@media.mit.edu> Message-ID: <50e8e4f60902030208x709c6b47ub93ce76a32254c3a@mail.gmail.com> On Mon, Feb 2, 2009 at 9:29 PM, Ian Eslick wrote: > Hi All, > > I was inspired the other day to come up with a minimal, quick-and- > dirty, all-lisp data store for Elephant based on cl-prevalence. Very nice ! And it is faster than BDB, great ! /Henrik Hjelte From patrice.lespagnol at obs-nancay.fr Tue Feb 3 14:15:20 2009 From: patrice.lespagnol at obs-nancay.fr (patrice.lespagnol at obs-nancay.fr) Date: Tue, 3 Feb 2009 14:15:20 -0000 (UTC) Subject: [elephant-devel] Lisp-only data store (prototype) Message-ID: For information, in case you've not already look at it, beside cl-prevalence, there is also the bknr datastore code base that seems interesting. http://bknr.net/html/documentation.html Hope that can help and thanks for sharing your work on Elephant ! From eslick at media.mit.edu Tue Feb 3 16:12:35 2009 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 3 Feb 2009 11:12:35 -0500 Subject: [elephant-devel] Lisp-only data store (prototype) In-Reply-To: References: Message-ID: The BKNR store is very interesting! It is a solution on-par with Elephant, but makes different commitments and tradeoffs. To utilize their infrastructure and make it compatible with Elephant's MOP implementation, I'd probably have to fork the tree and split out the appropriate functionality. I think that the Elephant MOP and transaction model is easier to work with and subject to fewer 'gotchas'. Elephant provides ACID guarantees to all primitive operations both inside and outside of transaction wrappers. Transactions ensure the atomicity of a set of operations (e.g. the account transfer example) and provide increased performance by reducing disk sync operations and lock grabbing/ releasing. I actually have a sketch of a pure prevalence solution in my contrib directory which is very similar to the BKNR approach, but using the Elephant MOP so it's compatible with other data stores. One thing I hadn't thought of before is leveraging their serializer and log file code base rather than re-writing my own which I had started some time ago. The design issue I was stuck on at the time, some of which is documented in the elephant-devel archives, is taking a BDB-like fine- grained locking approach to enable high degrees of concurrency or a copy-on-write / versioning approach similar to Rucksack. I won't have time for quite awhile, unfortunately, to get back to a full prevalence or on-disk btree, lisp-only solution. I'm happy to support someone who wants to take on the bulk of this work, however. The cl-prevalence solution actually provides this same set of properties, albeit with a noticeable performance hit. There are many opportunities to improve the performance of the CLP data store prototype, and of course it is not yet feature complete. It is fairly small, so I don't think it would be too much work for others to contribute to and could be made feature complete with some concerted effort in a week or two. FYI - Elephant's commitment to export a btree interface is a complicating factor for data store implementation. My thought is that we should probably hide this over time and export a higher level abstraction that eschews cursors and other low level details in favor of a table abstraction that includes pre-order and post-order traversal options via mapping abstractions and/or generators. This wouldn't be hard to do for 1.0, and we could maintain the btree abstraction as an add-on for those who depend on it. How many people currently directly use the Btree/cursor interface? Ian On Feb 3, 2009, at 9:15 AM, patrice.lespagnol at obs-nancay.fr wrote: > For information, in case you've not already look at it, beside > cl-prevalence, there is also the bknr datastore code base that seems > interesting. > http://bknr.net/html/documentation.html > Hope that can help and thanks for sharing your work on Elephant ! > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Tue Feb 3 16:56:47 2009 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 3 Feb 2009 11:56:47 -0500 Subject: [elephant-devel] Windows/SBCL bug (AND FIX) when compiling DLLs (WAS: partial fix) In-Reply-To: <42c0ab790902021545y7270723by3b4a9dd61bb802c0@mail.gmail.com> References: <42c0ab790901080114l4821f60fo8051b7ffca6d55b9@mail.gmail.com> <42c0ab790902021545y7270723by3b4a9dd61bb802c0@mail.gmail.com> Message-ID: All outstanding patches pushed, and I also updated the DLL link to include Elliott's kindly supplied pre-built DLLs for Windows. Ian On Feb 2, 2009, at 6:45 PM, Elliott Slaughter wrote: > Just wondering if there is any chance this patch will be reviewed > and/or integrated soon? > > Thanks. > > On Thu, Jan 8, 2009 at 1:14 AM, Elliott Slaughter > wrote: > Hi, > > On Mon, Dec 15, 2008 at 11:03 PM, Elliott Slaughter > wrote: > I just encountered a peculiarity with the perform method for > elephant-c-source files, which causes a clean build of elephant- > unstable on Windows/SBCL to fail. > > Attached is a patch which fixes this problem in a better way, and > safely deals with different drive letters and paths with spaces. > > Enjoy :-) > > -- > Elliott Slaughter > > "Any road followed precisely to its end leads precisely nowhere." - > Frank Herbert > > > > -- > Elliott Slaughter > > "Any road followed precisely to its end leads precisely nowhere." - > Frank Herbert > win32_sbcl_.dpatch>_______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Tue Feb 3 16:57:51 2009 From: eslick at media.mit.edu (Ian Eslick) Date: Tue, 3 Feb 2009 11:57:51 -0500 Subject: [elephant-devel] elephant-1.0 on Lispworks+win32; bdb; postmodern In-Reply-To: <1537352222@web.de> References: <1537352222@web.de> Message-ID: Hi Frank, Does the latest source tree resolve this bug? Ian On Jan 28, 2009, at 2:08 PM, Frank Schorr wrote: >> -----Urspr?ngliche Nachricht----- >> Von: "Ian Eslick" > >> >> Did this happen on the first open-controller or somewhere in the >> middle of the tests? >> > > I can't tell. > But I tried this: > > (asdf:operate 'asdf:load-op :elephant) > (asdf:operate 'asdf:load-op :ele-bdb) > (setf *db-spec* '(:BDB "c:/temp/testdb/")) > (ele:open-store *db-spec*) > > [The bdb files are created in the directory. Hope this helps] > > Error: The slot ELEPHANT::SCHEMA-NAME-INDEX is unbound in the object > # (an instance of class > #). > 1 (continue) Try reading slot ELEPHANT::SCHEMA-NAME-INDEX again. > 2 Specify a value to use this time for slot ELEPHANT::SCHEMA-NAME- > INDEX. > 3 Specify a value to set slot ELEPHANT::SCHEMA-NAME-INDEX to. > 4 (abort) Abort job 4 :(BIND-STANDARD-STREAMS-AND-EXECUTE > # > 21CE1B7F> (FUNCALL-BACKGROUND-JOB-AUX # PANE 217334BB> BACKGROUND-REGION-EVAL (# # # # T NIL NIL))) > > Type :b for backtrace, :c