From killerstorm at newmail.ru Sat Dec 8 11:08:32 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 8 Dec 2007 13:08:32 +0200 Subject: [elephant-devel] db-postmodern: performance considerations Message-ID: hello unfortunately rewrite of pm-cursor implementation is postponed for now, it turned to be more complex than it seemed originally. and i do not have test cases anyway.. however i've made a patch that improves performance of get-instances-by-value (i've sent it to Henrik so it should be available when he'll synchronize with upstream). so i'll describe profiling results here -- what's fast and what's not with postmodern backend. 1. individual operation in btree -- setting and getting value -- are converted directly into SQL queries, and so they are quite fast (as query in indexed SQL table can be). however, there's considerable client/server communication overhead (system calls, TCP/IP stack..), so making tons of individual sets/gets is not very fast. however, it's possible to cache get operations on client with postmodern backend. two caching options are available -- per-transaction-cache caches only inside one transaction. global-sync-cache caches data between transactions, synchronizing it with other working instances -- it tracks changes and invalidates cache entries accordingly. synchronization brings some overhead, and thus it's good only for certain types of workload (mostly-read). it's possible to implement global-cache for single-instance mode, that will do no synchronization, but this mode is not implemented because db-postmodern focuses of safety of use with multiple instances working with database simultaneously. get-instance-by-value, after a patches suggested by Alain, uses btree get-value, and thus is cached too. there's gotcha when using caches with large data sets -- cache entries doesn't get garbage collected, so it's possible to run out of memory. it's possible to use weak hash table in this case (see make-backend-cache in pm-cache, no configuration option), but i don't know how good caching will be with it. patch for some smarter solution is welcome. 2. get-instances-by-value, with my specialized map-index implementation, uses SQL query to retrieve data directly, so it should be pretty fast. however it doesn't use caching. it doesn't uses cursor, so all instances are always returned, it may be a problem if there's a lot of instances for some key. 3. cursors: in general, they do not work very well for now. first of all, they do not always return values in order as specified by "lisp sorter". moreover, if you do not have keys of same type (either all integers or strings) you get a random order. (with integers and strings you get order according to SQL comparison rules). we are not going to fix this issue (besides, maybe, allowing NILs to be mixed with integers or strings safely), since we believe it's more important to support good performance in practical cases (having all keys of same type). we could implement some emulation mode that will retrieve all data and sort it on lisp side, but probably it's better to use CLSQL backend if "lisp sort" ordering is a requirement of an application. then, probably the only thing that cursors are good at is iterating all the sequence from start to the end. iterating some limited set of values from start *almost* works fine. to do it efficiently on large table it's required to configure PostgreSQL -- disable hashjoin and mergejoin, otherwise it thinks that it's faster to process whole table rather than doing it incrementally. processing table with 10000 items costs 60-70 milliseconds. (additionally i suspect some small patch to db-postmodern is required to build correct indices for cursors to work incrementally). iterating from end doesn't work good -- it scans through all table (probably even twice) due to bug in cursor implementation. this should be more-or-less easily fixable, though. cursor-set performance depends on how far is key you search for from the start of the table. (!). it's counting on postgresql side, though, so for 10000 items table it should take about 70 msecs in worst case. multiple cursor-set calls will get equally slow all. i have some ideas how to fix cursor implementation so it will not depend on size of table (to the extent PostgreSQL does not depend, of course), but i don't know when i'll have a chance to implement it. as i've mentioned, i've currently do not even have a practical test case where cursor iteration is used. 4. get-instances-by-range uses cursors, and so it inherits all it's problems 5. psets: default implementation of psets makes instance of btree for each of them. in db-postmodern btree has it's own SQL table, so it's not a good idea to have thousands of psets. although they might be quite useful to have in big amounts, so probably we'll invent something better in future for them. with best regards, Alex 'killer_storm' Mizrahi. From killerstorm at newmail.ru Sat Dec 8 11:16:27 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 8 Dec 2007 13:16:27 +0200 Subject: [elephant-devel] Re: db-postmodern: performance considerations References: Message-ID: AM> so i'll describe profiling results here -- what's fast and what's not AM> with postmodern backend. probably i also need some kind of conclusion.. so, db-postmodern seems to work fine when it's used with persistent instances of classes, and only basic queries (get-instance-by-value, get-instances-by-value) are used to retrieve them. but when some more sophisticated queries are used it can s... be suboptimal :). From leslie.polzer at gmx.net Sat Dec 8 11:24:50 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 8 Dec 2007 12:24:50 +0100 (CET) Subject: [elephant-devel] Live backup Message-ID: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org> How would I go about doing a live backup of an Elephant-persistent set of objects? I suppose just copying the Berkeley DB would result in inconsistencies... Leslie From eslick at csail.mit.edu Sat Dec 8 13:38:18 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 8 Dec 2007 08:38:18 -0500 Subject: [elephant-devel] Live backup In-Reply-To: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org> References: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org> Message-ID: <55A1D29A-16FE-4947-B91D-21AC7C230DB2@csail.mit.edu> There really isn't a good solution to this that I'm aware of other than a stop and copy. The easiest would be to halt all transactions, do a file system copy, then continue. That might only take a handful of seconds depending on the size of your DB and speed of your system. There are more complex ways to do this that are driven by your application model that are similar to traditional GC. Implement a copy-on-write policy in your transactions and a create-new-in-new policy for object creation and just copy all the objects from one store to another. The old store is now your snapshot, and the new store has all your live data. This would, however, break all the nice indexing stuff we have (get-instances-by-value) since the class (metaclass instance) would be pointing at the old store. We haven't really thought through or provided support for this second case, but we're open to suggestions on how this could/should be done! Ian On Dec 8, 2007, at 6:24 AM, Leslie P. Polzer wrote: > > How would I go about doing a live backup of an Elephant-persistent > set of objects? I suppose just copying the Berkeley DB would result > in inconsistencies... > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Sat Dec 8 13:48:54 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 8 Dec 2007 08:48:54 -0500 Subject: [elephant-devel] db-postmodern: performance considerations In-Reply-To: References: Message-ID: <204AE769-9543-451C-9DF3-9313D753E719@csail.mit.edu> Hi Alex & all, 1) In general we seem to have data stores that have different sweet spots and different optimal uses of an API. We probably should come up with a Matrix of features and suggested uses for each of them. This includes what types are in lisp-sort-order and which are not for each data store. We should also have a spec of the common/safe features. 2) In retrospect it was a mistake to expose the cursor API to the users, but Elephant started as a low-level interface to BDB with some minimal support for persistent objects and has grown from there so at the time it made sense. I recommend we consider deprecating the cursor API for user use and find the use cases that people care about and implement a higher level API to them (abstract datastructures, a query language, etc). Then we drop support for the cursor API (but document what works for those who want to shoot themselves in the foot). The map API is intended to provide a slightly higher level interface to cursors so that you can imagine a traversal of a large set, only the current element of which 'must' be in memory so you can reasonably expect GC's to occur while doing a single map operation. What prospects are there for having a reasonable map implementation in postmodern? 3) get-instances-by-range should be implementable for integers and strings on postmodern, but should flag a datastore-specific error if a different type is used. Then you should be able to do a SQL query that returns all instances > some value and < another value. That shouldn't be hard to implement. 4) btrees are lightweight in BDB, but heavy in SQL - sounds like psets need a postmodern specific implementation. You need a table that implements a many-to-many relation between pset ids and object ids. Thanks for all the hard work Alex! Cheers, Ian PS - Has anyone validated my new default map-index under postmodern, or is that moot now that you have a specialized version? On Dec 8, 2007, at 6:08 AM, Alex Mizrahi wrote: > hello > > unfortunately rewrite of pm-cursor implementation is postponed for > now, it > turned to be more complex than it seemed originally. and i do not > have test > cases anyway.. > however i've made a patch that improves performance of > get-instances-by-value (i've sent it to Henrik so it should be > available > when he'll synchronize with upstream). > > so i'll describe profiling results here -- what's fast and what's > not with > postmodern backend. > > 1. individual operation in btree -- setting and getting value -- are > converted directly into SQL queries, and so they are quite fast (as > query in > indexed SQL table can be). however, there's considerable client/server > communication overhead (system calls, TCP/IP stack..), so making > tons of > individual sets/gets is not very fast. > > however, it's possible to cache get operations on client with > postmodern > backend. > two caching options are available -- per-transaction-cache caches only > inside one transaction. > global-sync-cache caches data between transactions, synchronizing it > with > other working instances -- it tracks changes and invalidates cache > entries > accordingly. synchronization brings some overhead, and thus it's > good only > for certain types of workload (mostly-read). > > it's possible to implement global-cache for single-instance mode, > that will > do no synchronization, but this mode is not implemented because > db-postmodern focuses of safety of use with multiple instances > working with > database simultaneously. > > get-instance-by-value, after a patches suggested by Alain, uses btree > get-value, and thus is cached too. > > there's gotcha when using caches with large data sets -- cache entries > doesn't get garbage collected, so it's possible to run out of > memory. it's > possible to use weak hash table in this case (see make-backend-cache > in > pm-cache, no configuration option), but i don't know how good > caching will > be with it. patch for some smarter solution is welcome. > > 2. get-instances-by-value, with my specialized map-index > implementation, > uses SQL query to retrieve data directly, so it should be pretty fast. > however it doesn't use caching. > > it doesn't uses cursor, so all instances are always returned, it may > be a > problem if there's a lot of instances for some key. > > 3. cursors: in general, they do not work very well for now. > > first of all, they do not always return values in order as specified > by > "lisp sorter". moreover, if you do not have keys of same type > (either all > integers or strings) you get a random order. (with integers and > strings you > get order according to SQL comparison rules). we are not going to > fix this > issue (besides, maybe, allowing NILs to be mixed with integers or > strings > safely), since we believe it's more important to support good > performance in > practical cases (having all keys of same type). we could implement > some > emulation mode that will retrieve all data and sort it on lisp side, > but > probably it's better to use CLSQL backend if "lisp sort" ordering is a > requirement of an application. > > then, probably the only thing that cursors are good at is iterating > all the > sequence from start to the end. > > iterating some limited set of values from start *almost* works fine. > to do > it efficiently on large table it's required to configure PostgreSQL -- > disable hashjoin and mergejoin, otherwise it thinks that it's faster > to > process whole table rather than doing it incrementally. processing > table > with 10000 items costs 60-70 milliseconds. (additionally i suspect > some > small patch to db-postmodern is required to build correct indices for > cursors to work incrementally). > > iterating from end doesn't work good -- it scans through all table > (probably > even twice) due to bug in cursor implementation. this should be more- > or-less > easily fixable, though. > > cursor-set performance depends on how far is key you search for from > the > start of the table. (!). it's counting on postgresql side, though, > so for > 10000 items table it should take about 70 msecs in worst case. > multiple > cursor-set calls will get equally slow all. > > i have some ideas how to fix cursor implementation so it will not > depend on > size of table (to the extent PostgreSQL does not depend, of course), > but i > don't know when i'll have a chance to implement it. as i've > mentioned, i've > currently do not even have a practical test case where cursor > iteration is > used. > > 4. get-instances-by-range uses cursors, and so it inherits all it's > problems > > 5. psets: default implementation of psets makes instance of btree > for each > of them. in db-postmodern btree has it's own SQL table, so it's not > a good > idea to have thousands of psets. although they might be quite useful > to have > in big amounts, so probably we'll invent something better in future > for > them. > > with best regards, Alex 'killer_storm' Mizrahi. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From Alain.Picard at memetrics.com Sun Dec 9 00:55:42 2007 From: Alain.Picard at memetrics.com (Alain Picard) Date: Sun, 09 Dec 2007 11:55:42 +1100 Subject: [elephant-devel] db-postmodern: performance considerations In-Reply-To: <204AE769-9543-451C-9DF3-9313D753E719@csail.mit.edu> (Ian Eslick's message of "Sat\, 8 Dec 2007 08\:48\:54 -0500") References: <204AE769-9543-451C-9DF3-9313D753E719@csail.mit.edu> Message-ID: <871w9wvg8h.fsf@memetrics.com> Ian Eslick writes: > What prospects are there for having a reasonable map implementation in > postmodern? Surely the prospects must be good; it's not very different from what Lispworks' CommonSQL does with MAP-QUERY. > 4) btrees are lightweight in BDB, but heavy in SQL - sounds like psets > need a postmodern specific implementation. You need a table that > implements a many-to-many relation between pset ids and object ids. This is indeed a problem which bit me, and I find I need to spend more time than I had hoped looking "under the hood" to see if my decisions at the classes/api level map onto horrible implementations. > Thanks for all the hard work Alex! I would certainly to join in the thanks. So thanks! -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From killerstorm at newmail.ru Sun Dec 9 02:15:33 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 9 Dec 2007 04:15:33 +0200 Subject: [elephant-devel] Re: db-postmodern: performance considerations References: <204AE769-9543-451C-9DF3-9313D753E719@csail.mit.edu> Message-ID: IE> 2) In retrospect it was a mistake to expose the cursor API to the IE> users, but Elephant started as a low-level interface to BDB with some IE> minimal support for persistent objects and has grown from there so at IE> the time it made sense. I recommend we consider deprecating the IE> cursor API for user use and find the use cases that people care about IE> and implement a higher level API to them (abstract datastructures, a IE> query language, etc). Then we drop support for the cursor API (but IE> document what works for those who want to shoot themselves in the IE> foot). i know at least one usage pattern that can be done via cursor API, but it's unlikely that map can support this even if we improve it. say we need 10 latest items. we can get them via map, passing it a tricky function that will abort enumeration as soon as it will collect 10 items. but what if need 10 latest items of different types (coming from different indices) -- say apples and oranges? with cursors it's straightforward to do this, but with map it's quite tricky if doable at all IE> The map API is intended to provide a slightly higher level interface IE> to cursors so that you can imagine a traversal of a large set, only IE> the current element of which 'must' be in memory so you can reasonably IE> expect GC's to occur while doing a single map operation. IE> What prospects are there for having a reasonable map implementation in IE> postmodern? to implement map in "reasonable" way (so it will be interruptible and will not demand all dataset being retrieved to fit in memory) we need kind of cursors, so it's simplier just to implement good cursors, and we already have a map implementation :). the problem with current db-postmodern cursor implementation is that it fetches data via SQL cursors, and assumtion that they work fast was not correct. i have an idea how to do fetching efficiently: we have current key/value in cursor, and when we need next (for example) we execute such query: SELECT key, value FROM btree11 WHERE (key > our_key) ORDER BY key LIMIT 1 ; for duplicate-less btrees SELECT key, value FROM btree11 WHERE (key > our_key) OR ((key = our_key) AND (value > our_value)) ORDER BY key, value LIMIT 1 ; for btree-index i've tested queries like this, seem to work pretty fast in PostgreSQL. of course we can fetch next 10 into cache, for example, to minimize communication overhead. and this seems to fit well with the rest of implementation, so only code that is fetching data from backend needs to be changed. well, i think i'll just try to implement this.. IE> Thanks for all the hard work Alex! actually hard work was done by Henrik, so all thanks for postmodern backend should go to him. i've just polished it a little.. IE> PS - Has anyone validated my new default map-index under postmodern, IE> or is that moot now that you have a specialized version? only value-set-p case is specialized, other cases use generic implementation. if there are tests for this (grep finds some), it passes them. From Alain.Picard at memetrics.com Sun Dec 9 05:41:56 2007 From: Alain.Picard at memetrics.com (Alain Picard) Date: Sun, 09 Dec 2007 16:41:56 +1100 Subject: [elephant-devel] Mixing "logical" layers in an elephant application Message-ID: <87tzmstoez.fsf@memetrics.com> Dear Elephant developers, I'm trying to develop an application using Elephant, and I think I've settled on the POSTMODERN back end. Now, part of the application maps very naturally in what Elephant can do, and other parts of it map better onto raw SQL processing, where I wish to use POSTMODERN more or less directly. As a toy example, consider this: ;; in postmodern specific file (defun map-visits (fn thing-id &key (from 0) (to 9999999999) collect) (let ((results '())) (flet ((collector (thing) (push (funcall fn thing) results))) (let ((fn (if collect #'collector fn))) (doquery (:select thing-id :from :visit :where (:and (:= thing-id id) (:< from 'timestamp) (:<= 'timestamp to))) (arg) (funcall fn arg)))) results)) ;; Later, in an elephant specific file (defmethod process (thing &key from to) (let ((contribs (make-hash-table))) (flet ((compute-something (arg) (incf (gethash arg contribs 0)))) (ensure-transaction () (map-visits #'add-contribution (person-name fan) :from from :to to))) contribs)) The first function knows how to map over "visits", which are things inserted into a postgresql table by a process other than my app. It's a very simple SQL table. But it relates semantically to objects held within the elephant database; e.g. the second function, PROCESS-THING wants to map over all the visits of a THING, and do some elephantine business with each visit, but doesn't want to concern itself with how VISITS are implemented. When I tried the above, it blew up when I added the ENSURE-TRANSACTION clause; because all of a sudden the DOQUERY found itself querying on the "wrong" database. (I'm assuming ENSURE-TRANSACTION binds something like a *DATABASE* var which got mistakenly inherited by the postmodern layer) This led me to thinking about what other unknown effects/surprises may be lurking for me; e.g. how transactions in one "world" will interact with another one, etc. I have read the manual, including the terse section "4.12 Multi-repository Operation", but what I'm attempting may be even hairier. Inspecting the code, I'm currently tempted to go hack up something based on DB-POSTMODERN::CONTROLLER-CONNECTION-FOR-THREAD to "marry" the two DB layers, but I thought I'd ask first if anyone else has considered these sorts of issues and if there is a "recommended" way of proceeding in a case such as this. I guess the first thing I'd try is to bind something way up in the elephant call stack which would cause all lower level "raw" postmodern calls to act on the same DB that the *STORE-CONTROLLER* is currently using. Any thoughts or comments will be deeply appreciated. Alain Picard -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From killerstorm at newmail.ru Sun Dec 9 11:39:08 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 9 Dec 2007 13:39:08 +0200 Subject: [elephant-devel] Re: Mixing "logical" layers in an elephant application References: <87tzmstoez.fsf@memetrics.com> Message-ID: AP> When I tried the above, it blew up when I added the ENSURE-TRANSACTION AP> clause; because all of a sudden the DOQUERY found itself querying AP> on the "wrong" database. (I'm assuming ENSURE-TRANSACTION binds AP> something like a *DATABASE* var which got mistakenly inherited by AP> the postmodern layer) so you have two postmodern connections? there are two layers in POSTMODERN: postmodern itself uses special variable *database* to control which connection is used, but in lower level cl-postgres connection is passed explicitly. elephant's backend uses both. it uses with-postmodern-conn macro when it's going to use it's own database. other times it uses cl-postgres layer and that shouldn't affect other uses (it uses it's own special variable *connection* and passes it to cl-postgres calls). it calls with-postmodern-conn in ensure-transaction (i'm not sure why it's there, but it is). so you need to fight-back your connection in your map-visits (or before you call it, but inside elephant's ensure-transaction). even if you call elephant functions from it, it should automatically switching connections and it should "just work". if it doesn't, it's a bug, and with-postmodern-conn macro should be inserted somewhere. alternatively you can use cl-postgres, or make your own macros that both allow fancy syntax and explicit connection specification.. i think that besides connection nothing can be messed with postmodern -- if you do two connections they are totally independent. but i'm afraid you're first human on the earth who uses both elephant/db-postmodern and postmodern combination simultaneously, so nobody really knows.. :) From killerstorm at newmail.ru Sun Dec 9 16:25:43 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 9 Dec 2007 18:25:43 +0200 Subject: [elephant-devel] Mixing "logical" layers in an elephant application References: <87tzmstoez.fsf@memetrics.com> Message-ID: <000701c83a80$25f8b3e0$720210ac@killer> AP> When I tried the above, it blew up when I added the ENSURE-TRANSACTION clause; because all of a sudden the DOQUERY found itself querying on the "wrong" database. (I'm assuming ENSURE-TRANSACTION binds something like a *DATABASE* var which got mistakenly inherited by the postmodern layer) so you have two postmodern connections? there are two layers in POSTMODERN: postmodern itself uses special variable *database* to control which connection is used, but in lower level cl-postgres connection is passed explicitly. elephant's backend uses both. it uses with-postmodern-conn macro when it's going to use it's own database. other times it uses cl-postgres layer and that shouldn't affect other uses (it uses it's own special variable *connection* and passes it to cl-postgres calls). it calls with-postmodern-conn in ensure-transaction (i'm not sure why it's there, but it is). so you need to fight-back your connection in your map-visits (or before you call it, but inside elephant's ensure-transaction). even if you call elephant functions from it, it should automatically switching connections and it should "just work". if it doesn't, it's a bug, and with-postmodern-conn macro should be inserted somewhere. alternatively you can use cl-postgres, or make your own macros that both allow fancy syntax and explicit connection specification.. i think that besides connection nothing can be messed with postmodern -- if you do two connections they are totally independent. but i'm afraid you're first human on the earth who uses both elephant/db-postmodern and postmodern combination simultaneously, so nobody really knows.. :) (seems like gmane have ate my post, re-sending it directly. apologies if it will be duplicated) From read at robertlread.net Sun Dec 9 23:06:11 2007 From: read at robertlread.net (Robert L. Read) Date: Sun, 09 Dec 2007 17:06:11 -0600 Subject: [elephant-devel] Mixing "logical" layers in an elephant application In-Reply-To: <87tzmstoez.fsf@memetrics.com> References: <87tzmstoez.fsf@memetrics.com> Message-ID: <1197241571.5370.189.camel@penguin.yourdomain.com> Dear Alain, I think you have analyzed the problem very well. As you say, in the postmodern codebase in the file query.lisp, the macro "doquery" utilizes the *database* variable with no obvious way to override or specify it. I would have to study some more to be absolutely sure, but it does seem quite likely that the other postmodern macros that are utilized by elephant and mention the same variable probably make it difficult to interleave postmodern calls of any kind to different repositories, whether elephant is involved or not, and of course the ele-postmodern code also ends up setting that variable, invisibly to the user. Ian and I once had to deal with the same issue in Elephant's use of CL-SQL. The basic solution is to always float parameters up (at least optionally), so that instead of writing: (doquery q (arg) (blah-blah arg)), one at least has the option of writing: (doquery q (arg) (blah-blah arg) :with-this-connection UNEXPECTEDCONNECTION) However, this is a non-trivial change to the postmodern code. In the testmigrations.lisp file in the /tests directory in the elepahnt codebase, you will find the following idiom, which can be used to sequester a globabl value, change it, and restore it. (let ((old-store *store-controller*) (*store-controller* nil)) (unwind-protect (do-a-bunch of stuff that sets *store-controller*) (progn (setq *store-controller* old-store))) ) It is possible that by doing something similar to the postmodern::*database value (instead of the *store-controller*), you could make your call out to "doquery" in the sample code you mention, and by restoring *database*, be able to carry on. (Of course, you should not have to do this, this is clearly a bug at some level. It is not entirely clear to me where it should be fixed. I would have to hear from Henrik or Alex, or study at great length, to answer your deeper question: what other surprises might await multi-connection operation via postmodern. I hope Henrik will read this and answer that question.) The long term solution to the problem (whether provided by postmodern, ele-postmodern, or the core of elephant, I'm not sure) is to allow fully specification of which connnection to use in every operation (while trying to use intelligent defaults, so that the most calls do not require it to be specified.) So, if you can forgive my ignorance of postmodern, the question for Henrik are: 1) Can postmodern interleave calls to different transactions, and if so how? 2) If it can't, is manipulating *database* a reasonable short-term solution? Is there other state (caches, for example), that might also have to be "swizeled"? On Sun, 2007-12-09 at 16:41 +1100, Alain Picard wrote: > Dear Elephant developers, > > I'm trying to develop an application using Elephant, and > I think I've settled on the POSTMODERN back end. > > Now, part of the application maps very naturally in what Elephant can > do, and other parts of it map better onto raw SQL processing, > where I wish to use POSTMODERN more or less directly. As a > toy example, consider this: > > ;; in postmodern specific file > (defun map-visits (fn thing-id &key (from 0) (to 9999999999) collect) > (let ((results '())) > (flet ((collector (thing) > (push (funcall fn thing) results))) > (let ((fn (if collect #'collector fn))) > (doquery (:select thing-id > :from :visit > :where (:and (:= thing-id id) > (:< from 'timestamp) > (:<= 'timestamp to))) > (arg) > (funcall fn arg)))) > results)) > > ;; Later, in an elephant specific file > (defmethod process (thing &key from to) > (let ((contribs (make-hash-table))) > (flet ((compute-something (arg) > (incf (gethash arg contribs 0)))) > (ensure-transaction () > (map-visits #'add-contribution (person-name fan) :from from :to to))) > contribs)) > > The first function knows how to map over "visits", which are things > inserted into a postgresql table by a process other than my app. > It's a very simple SQL table. But it relates semantically to objects > held within the elephant database; e.g. the second function, > PROCESS-THING wants to map over all the visits of a THING, and do > some elephantine business with each visit, but doesn't want to concern > itself with how VISITS are implemented. > > When I tried the above, it blew up when I added the ENSURE-TRANSACTION > clause; because all of a sudden the DOQUERY found itself querying > on the "wrong" database. (I'm assuming ENSURE-TRANSACTION binds > something like a *DATABASE* var which got mistakenly inherited by > the postmodern layer) > > This led me to thinking about what other unknown effects/surprises > may be lurking for me; e.g. how transactions in one "world" will > interact with another one, etc. I have read the manual, including the > terse section "4.12 Multi-repository Operation", but what I'm attempting > may be even hairier. > > Inspecting the code, I'm currently tempted to go hack up something > based on DB-POSTMODERN::CONTROLLER-CONNECTION-FOR-THREAD to > "marry" the two DB layers, but I thought I'd ask first if anyone > else has considered these sorts of issues and if there is a > "recommended" way of proceeding in a case such as this. I guess > the first thing I'd try is to bind something way up in the elephant > call stack which would cause all lower level "raw" postmodern calls > to act on the same DB that the *STORE-CONTROLLER* is currently > using. > > Any thoughts or comments will be deeply appreciated. > > Alain Picard > > > From henrik at evahjelte.com Tue Dec 11 09:20:36 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Tue, 11 Dec 2007 10:20:36 +0100 Subject: [elephant-devel] Live backup In-Reply-To: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org> References: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60712110120t14241792qe18c71e8164ab03c@mail.gmail.com> On Dec 8, 2007 12:24 PM, Leslie P. Polzer wrote: > > How would I go about doing a live backup of an Elephant-persistent > set of objects? I suppose just copying the Berkeley DB would result > in inconsistencies... I have made a basic export facility, which can dump an elephant database to sexp format, and restore it. It can be used to take snapshots of your database. I use it quite often to migrate databases: darcs get http://common-lisp.net/project/grand-prix/darcs/gp-export The biggest negative feature is that the whole database must fit into memory, but when our database becomes bigger than that I will fix it.. Also, if you use a database backend (clsql/postmodern) the database will most often have a live backup feature built in. At least postgresql has. /Henrik -------------- next part -------------- An HTML attachment was scrubbed... URL: From killerstorm at newmail.ru Tue Dec 11 12:43:46 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 11 Dec 2007 14:43:46 +0200 Subject: [elephant-devel] Re: Live backup References: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org> <55A1D29A-16FE-4947-B91D-21AC7C230DB2@csail.mit.edu> Message-ID: IE> policy for object creation and just copy all the objects from one IE> store to another. if bdb has serializable isolation level be default, woudn't it be enough just to copy data within transaction? From eslick at csail.mit.edu Tue Dec 11 14:28:08 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 11 Dec 2007 09:28:08 -0500 Subject: [elephant-devel] Re: Live backup In-Reply-To: References: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org> <55A1D29A-16FE-4947-B91D-21AC7C230DB2@csail.mit.edu> Message-ID: <17E6F21A-D792-4485-8027-83D907DBBE9E@csail.mit.edu> Copying all the data from a store isn't a big problem, copying a store without creating inconsistent state while the application continues to run is the challenge I thought the user was interested in. If you do it at the application level within a transaction (even if it did fit in memory) you would either implicitly halt all activity or constantly be aborted by any write operation anywhere in the db. While there is not an 'elephant' level way to do live backups, each underlying data store has it's own methods as pointed out by Henrik and Alex for the SQL cases. Berkeley DB also supports low-level hot backups if you know that your OS reads file pages atomically. (http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/archival.html ) For an example see the source code for db_hotbackup in the BDB distribution. Ian On Dec 11, 2007, at 7:43 AM, Alex Mizrahi wrote: > IE> policy for object creation and just copy all the objects from one > IE> store to another. > > if bdb has serializable isolation level be default, woudn't it be > enough > just to copy data within transaction? > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Tue Dec 11 15:41:52 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Tue, 11 Dec 2007 17:41:52 +0200 Subject: [elephant-devel] Re: Live backup References: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org><55A1D29A-16FE-4947-B91D-21AC7C230DB2@csail.mit.edu> <17E6F21A-D792-4485-8027-83D907DBBE9E@csail.mit.edu> Message-ID: <000001c83c0c$5aa55d40$720210ac@killer> IE> If you do it at the application level within a transaction (even if IE> it did fit in memory) you would either implicitly halt all activity or IE> constantly be aborted by any write operation anywhere in the db. ah, indeed, this way bdb works.. PostgreSQL uses MVCC by default, so reading activity doesn't block writting. documentation says you can also get MVCC in bdb via "snapshot isolation" -- it would implement copy-on-write on database level. see here: http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/read.html http://www.oracle.com/technology/documentation/berkeley-db/db/api_cxx/txn_begin.html DB_TXN_SNAPSHOT This transaction will execute with snapshot isolation. For databases with the DB_MULTIVERSION flag set, data values will be read as they are when the transaction begins, without taking read locks. Silently ignored for operations on databases with DB_MULTIVERSION not set on the underlying database (read locks are acquired). or that's what you meant when said "Implement a copy-on-write policy in your transactions"? IE> Berkeley DB also supports low-level hot backups if you know that your IE> OS reads file pages atomically. IE> (http://www.oracle.com/technology/documentation/berkeley-db/db/ref/tran IE> sapp/archival.html ) For an example see the source code for IE> db_hotbackup in the BDB distribution. that's probably preffered way indeed. From eslick at csail.mit.edu Tue Dec 11 16:29:13 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Tue, 11 Dec 2007 11:29:13 -0500 Subject: [elephant-devel] Re: Live backup In-Reply-To: <000001c83c0c$5aa55d40$720210ac@killer> References: <63809.88.73.243.12.1197113090.squirrel@mail.stardawn.org><55A1D29A-16FE-4947-B91D-21AC7C230DB2@csail.mit.edu> <17E6F21A-D792-4485-8027-83D907DBBE9E@csail.mit.edu> <000001c83c0c$5aa55d40$720210ac@killer> Message-ID: <9398CBA0-B3DD-4835-AF95-E6C82720F4A2@csail.mit.edu> On Dec 11, 2007, at 10:41 AM, Alex Mizrahi wrote: > IE> If you do it at the application level within a transaction > (even if > IE> it did fit in memory) you would either implicitly halt all > activity or > IE> constantly be aborted by any write operation anywhere in the db. > > ah, indeed, this way bdb works.. PostgreSQL uses MVCC by default, so > reading activity doesn't block writting. > > documentation says you can also get MVCC in bdb via "snapshot > isolation" -- it would implement copy-on-write on database level. > see here: > http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/read.html > http://www.oracle.com/technology/documentation/berkeley-db/db/api_cxx/txn_begin.html > DB_TXN_SNAPSHOT > This transaction will execute with snapshot isolation. For databases > with the DB_MULTIVERSION flag set, data values will be read as they > are when the transaction begins, without taking read locks. Silently > ignored for operations on databases with DB_MULTIVERSION not set on > the underlying database (read locks are acquired). > > or that's what you meant when said "Implement a copy-on-write > policy in your transactions"? I was thinking about something pretty similar, but at the application level. Perhaps my original response was too off-the-cuff. I think the only problem that may come up in the MVCC model is whether you run out of storage for keeping the write versions of committing transactions that start after the read operation snapshot was taken. It can take a long time to read the entire DB within a single transaction! > IE> Berkeley DB also supports low-level hot backups if you know that > your > IE> OS reads file pages atomically. > IE> (http://www.oracle.com/technology/documentation/berkeley-db/db/ref/tran > IE> sapp/archival.html ) For an example see the source code for > IE> db_hotbackup in the BDB distribution. > > that's probably preffered way indeed. Indeed. This puts the future transaction storage burden in the log files instead of a shared memory region which is much more scalable. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From rosssd at gmail.com Wed Dec 12 15:05:26 2007 From: rosssd at gmail.com (Sean Ross) Date: Wed, 12 Dec 2007 15:05:26 +0000 Subject: [elephant-devel] initialize-instance and deserialize Message-ID: <5bef28df0712120705o75641a74jbf8b455c09c19976@mail.gmail.com> Hi all, I've currently run into a problem using Elephant 0.9.1 and Lispworks 5.0.2 with the bdb backend. I have a user class on which I perform various initializations using initialize-instance, unfortunately my initialize instance method is being called when the instance is deserialised. Now while I can manually check whether the various slots have been initialized, it still feels slightly onerous and I was wondering if there was a way of working with Elephant or patching Elephant to use allocate-instance and do the necessary setup rather than using make-instance directly. cheers, sean. From eslick at csail.mit.edu Wed Dec 12 16:25:56 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 12 Dec 2007 11:25:56 -0500 Subject: [elephant-devel] initialize-instance and deserialize In-Reply-To: <5bef28df0712120705o75641a74jbf8b455c09c19976@mail.gmail.com> References: <5bef28df0712120705o75641a74jbf8b455c09c19976@mail.gmail.com> Message-ID: <53D1F725-F117-415E-89DE-678024AF5D46@csail.mit.edu> Hi Sean, This is a problem that Pierre was very interested in fixing. Pierre? I'm sure that something can be done to separate initial object creation from object re-instantiation on deserialization. As a quick hack, you can check for the validity of from-oid keyword argument to initialize-instance to find out if this is a new instance or a reconstituted one. You might trace the various generic functions to see the difference in behavior during new object creation and deserialization as a start if you want to figure this out yourself. I won't have any time to do any significant Elephant work until mid-to- late January, if then. Regards, Ian On Dec 12, 2007, at 10:05 AM, Sean Ross wrote: > Hi all, > I've currently run into a problem using Elephant 0.9.1 and Lispworks > 5.0.2 with the bdb backend. > > I have a user class on which I perform various initializations using > initialize-instance, unfortunately > my initialize instance method is being called when the instance is > deserialised. > > Now while I can manually check whether the various slots have been > initialized, it still feels slightly onerous > and I was wondering if there was a way of working with Elephant or > patching Elephant to > use allocate-instance and do the necessary setup rather than using > make-instance directly. > > cheers, > sean. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Wed Dec 12 18:42:12 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Wed, 12 Dec 2007 19:42:12 +0100 Subject: [elephant-devel] Some performance optimizations Message-ID: <50e8e4f60712121042ubbecca7j1b1e611eadf872db@mail.gmail.com> Attached is a bunch of patches that improves the postmodern backend, most important performance patches from Alex Mizrahi. I have tried them with the testsuite. Work on a new cursor implementation is in progress, that might come later. /Henrik -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: patches.darcs Type: application/octet-stream Size: 7496 bytes Desc: not available URL: From rosssd at gmail.com Fri Dec 14 14:41:12 2007 From: rosssd at gmail.com (Sean Ross) Date: Fri, 14 Dec 2007 14:41:12 +0000 Subject: [elephant-devel] initialize-instance and deserialize In-Reply-To: <53D1F725-F117-415E-89DE-678024AF5D46@csail.mit.edu> References: <5bef28df0712120705o75641a74jbf8b455c09c19976@mail.gmail.com> <53D1F725-F117-415E-89DE-678024AF5D46@csail.mit.edu> Message-ID: <5bef28df0712140641q650607b8xc6cdc8884d1ddbb1@mail.gmail.com> On Dec 12, 2007 4:25 PM, Ian Eslick wrote: > You might trace the various generic functions > to see the difference in behavior during new object creation and > deserialization as a start if you want to figure this out yourself. Right, after a couple of breakpoints and a little bit of backtracking I have a working version of this. I've added 2 new methods recreate-instance-using-class and recreate-instance. This is used by get-cached-instance and %deserialize which bypasses initialize-instance and calls appropriate methods for the persistent and persistent-object classes. Please find the attached patch. cheers, sean -------------- next part -------------- A non-text attachment was scrubbed... Name: initialization.patch Type: text/x-patch Size: 2536 bytes Desc: not available URL: From rosssd at gmail.com Fri Dec 14 15:07:44 2007 From: rosssd at gmail.com (Sean Ross) Date: Fri, 14 Dec 2007 15:07:44 +0000 Subject: [elephant-devel] Lispworks set-db-synch Message-ID: <5bef28df0712140707p2dd97b24se8e661b6b1c8c8c7@mail.gmail.com> Hi all, here's a small patch to swap the load order of classindex-utils.lisp and pset.lisp. This is required as Lispworks attempts to call set-db-synch when loading the pset fasl which causes it to fall over. cheers, sean -------------- next part -------------- A non-text attachment was scrubbed... Name: load-order.patch Type: text/x-patch Size: 571 bytes Desc: not available URL: From read at robertlread.net Fri Dec 14 20:05:16 2007 From: read at robertlread.net (Robert L. Read) Date: Fri, 14 Dec 2007 14:05:16 -0600 Subject: [elephant-devel] Lispworks set-db-synch In-Reply-To: <5bef28df0712140707p2dd97b24se8e661b6b1c8c8c7@mail.gmail.com> References: <5bef28df0712140707p2dd97b24se8e661b6b1c8c8c7@mail.gmail.com> Message-ID: <1197662716.7545.33.camel@penguin.yourdomain.com> Thank you for your patches. I will try to code review them next week, and will probably apply them then. On Fri, 2007-12-14 at 15:07 +0000, Sean Ross wrote: > Hi all, > here's a small patch to swap the load order of classindex-utils.lisp > and pset.lisp. > This is required as Lispworks attempts to call set-db-synch when > loading the pset fasl > which causes it to fall over. > > cheers, > sean > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From killerstorm at newmail.ru Sat Dec 15 16:47:28 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 15 Dec 2007 18:47:28 +0200 Subject: [elephant-devel] Re: Some performance optimizations References: <50e8e4f60712121042ubbecca7j1b1e611eadf872db@mail.gmail.com> Message-ID: > Work on a new cursor implementation is in progress, that > might come later. done, at least passes all tests.. of course this new implementation doesn't solve all issues, but at least cursor operations now do not slow down with table growth, to the extent PostgreSQL does not. btw, i've found such test: (test (map-btree-remove :depends-on (and btree-make btree-cursor map-btree)) (flet ((mapper (k v) (declare (ignore k)) (when (and (>= (slot1 v) 100) (< (slot1 v) 200)) (remove-current-kv)))) (map-btree #'mapper bt) seems kinda weird for me -- cursor-delete (which is called from remove-current-kv) -- leaves cursor uninitialized, so further cursor-next does cursor-first (looks like it's not documented for Elephant, but that's the way it works in bdb. so it starts again from start to reach these deleted values again.. won't it be better to define something like: (defun remove-current-kv () (unless *current-cursor* (error "Cannot call remove-current-kv outside of a map-btree or map-index function argument")) (multiple-value-bind (f k v) (when f (cursor-current *current-cursor*) (cursor-delete *current-cursor*) (cursor-set-range *current-cursor* k)))) ? btw tests helped me debugging thing a lot.. From killerstorm at newmail.ru Sat Dec 15 21:24:43 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 15 Dec 2007 23:24:43 +0200 Subject: [elephant-devel] Re: Some performance optimizations References: <50e8e4f60712121042ubbecca7j1b1e611eadf872db@mail.gmail.com> Message-ID: AM> of course this new implementation doesn't solve all issues, but at AM> least cursor operations now do not slow down with table growth, to the AM> extent PostgreSQL does not. also important note about cursor implementation, which makes it somewhat different from bdb cursor behaviour, -- it fetches batches of k/v pairs at once, to mitigate communication overhead (default batch size is 10). so, even if you delete values from btree, you have can still get them from cache of cursor. it can read from cache only on next/prev operations, operations like set fetch fresh data from database. effect of caching mostly will go away if you bind *cursor-window-size* to 1. also, duplicated cursor does not inherit cache, so cursor-duplicate can be used to flush caches. AM> won't it be better to define something like: AM> (defun remove-current-kv () AM> (multiple-value-bind (f k v) AM> (when f AM> (cursor-current *current-cursor*) AM> (cursor-delete *current-cursor*) AM> (cursor-set-range *current-cursor* k)))) obviously that won't work.. with db-postmodern we can use the fact that cursor works correctly regardless of if current key/value is still in db. so we can do: (defun remove-current-kv () (cursor-delete (cursor-duplicate *current-cursor*))) i don't know if it works in other backends.. or maybe they do not make cursor uninitialized on delete and problem is non-existent? without this property, i'm afraid it's not possible to implement remove-current-kv without using details of current iteration operation. From eslick at csail.mit.edu Sun Dec 16 02:53:02 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 15 Dec 2007 21:53:02 -0500 Subject: [elephant-devel] Re: Some performance optimizations In-Reply-To: References: <50e8e4f60712121042ubbecca7j1b1e611eadf872db@mail.gmail.com> Message-ID: On Dec 15, 2007, at 4:24 PM, Alex Mizrahi wrote: > AM> of course this new implementation doesn't solve all issues, but at > AM> least cursor operations now do not slow down with table growth, > to the > AM> extent PostgreSQL does not. > > also important note about cursor implementation, which makes it > somewhat > different from bdb cursor behaviour, -- it fetches batches of k/v > pairs at > once, to mitigate communication overhead (default batch size is 10). > so, > even if you delete values from btree, you have can still get them > from cache > of cursor. it can read from cache only on next/prev operations, > operations > like set fetch fresh data from database. > effect of caching mostly will go away if you bind *cursor-window- > size* to 1. > also, duplicated cursor does not inherit cache, so cursor-duplicate > can be > used to flush caches. > > AM> won't it be better to define something like: > > AM> (defun remove-current-kv () > AM> (multiple-value-bind (f k v) > AM> (when f > AM> (cursor-current *current-cursor*) > AM> (cursor-delete *current-cursor*) > AM> (cursor-set-range *current-cursor* k)))) > > obviously that won't work.. > with db-postmodern we can use the fact that cursor works correctly > regardless of if current key/value is still in db. so we can do: This may be a silly question, but can't you just flush the cache on delete so it has to fetch the window again? > (defun remove-current-kv () > (cursor-delete (cursor-duplicate *current-cursor*))) > > i don't know if it works in other backends.. or maybe they do not make > cursor uninitialized on delete and problem is non-existent? > > without this property, i'm afraid it's not possible to implement > remove-current-kv without using details of current iteration > operation. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From Alain.Picard at memetrics.com Sat Dec 15 00:11:34 2007 From: Alain.Picard at memetrics.com (Alain Picard) Date: Sat, 15 Dec 2007 11:11:34 +1100 Subject: [elephant-devel] Re: Mixing "logical" layers in an elephant application In-Reply-To: (Alex Mizrahi's message of "Sun\, 9 Dec 2007 13\:39\:08 +0200") References: <87tzmstoez.fsf@memetrics.com> Message-ID: <877ijgakax.fsf@memetrics.com> "Alex Mizrahi" writes: > AP> When I tried the above, it blew up when I added the > ENSURE-TRANSACTION > AP> clause; because all of a sudden the DOQUERY found itself querying > AP> on the "wrong" database. (I'm assuming ENSURE-TRANSACTION binds > AP> something like a *DATABASE* var which got mistakenly inherited by > AP> the postmodern layer) > > so you have two postmodern connections? At least. By which I mean that I want to use connection pools, both at the elephant and "raw" postmodern levels. Adding a WITH-CONNECTION in the judicious place: (defun map-visits (fn thing-id &key (from 0) (to 9999999999) collect) (let ((results '())) (flet ((collector (thing) (push (funcall fn thing) results))) (let ((fn (if collect #'collector fn))) (pomo:with-connection ("otherdb" "postgres" "" "localhost" :pooled-p t) (doquery (:select thing-id :from :visit :where (:and (:= thing-id id) (:< from 'timestamp) (:<= 'timestamp to))) (arg) (funcall fn arg))))) results)) Does indeed allow the calling code to MAP-VISIT on "otherdb" while the FN being passed in is a closure which uses elephant to perform operations on "elephantdb" in the meantime. It _seems_ to me that if FN throws, it's transaction should unroll, and since MAP-VISITS is just doing a select, it doesn't need to be in a transaction, but if it were, it would also unroll correctly. I'll go with this for now and report problems if I find any. Thanks for the advice! --Alain -- Please read about why Top Posting is evil at: http://en.wikipedia.org/wiki/Top-posting and http://www.dickalba.demon.co.uk/usenet/guide/faq_topp.html Please read about why HTML in email is evil at: http://www.birdhouse.org/etc/evilmail.html From killerstorm at newmail.ru Sun Dec 16 10:04:59 2007 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 16 Dec 2007 12:04:59 +0200 Subject: [elephant-devel] Re: Some performance optimizations References: <50e8e4f60712121042ubbecca7j1b1e611eadf872db@mail.gmail.com> Message-ID: IE> This may be a silly question, but can't you just flush the cache on IE> delete so it has to fetch the window again? on delete it not only flushes cache, but makes cursor uninitialized, as required by documentation: - Generic Function: elephant:cursor-delete cursor Delete by cursor. The cursor is at an invalid position, and uninitialized, after a successful delete. so, next cursor-next will be actually cursor-first and start iteration from start. or course we can just ignore documentation and leave cursor where it is, so cursor-next will continute iteration. but what's about other backends? From read at robertlread.net Mon Dec 17 21:47:58 2007 From: read at robertlread.net (Robert L. Read) Date: Mon, 17 Dec 2007 15:47:58 -0600 Subject: [elephant-devel] Some performance optimizations In-Reply-To: <50e8e4f60712121042ubbecca7j1b1e611eadf872db@mail.gmail.com> References: <50e8e4f60712121042ubbecca7j1b1e611eadf872db@mail.gmail.com> Message-ID: <1197928078.12657.118.camel@penguin.yourdomain.com> I was also green on the tests based on this set of patches, so I have applied it to the main repository. Anyone who used "darcs pull" to get the latest changes should receive the benefit of these patches. On Wed, 2007-12-12 at 19:42 +0100, Henrik Hjelte wrote: > Attached is a bunch of patches that improves the > postmodern backend, most important performance > patches from Alex Mizrahi. I have tried them with the testsuite. > Work on a new cursor implementation is in progress, that > might come later. > > /Henrik > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Mon Dec 17 21:49:52 2007 From: read at robertlread.net (Robert L. Read) Date: Mon, 17 Dec 2007 15:49:52 -0600 Subject: [elephant-devel] Lispworks set-db-synch In-Reply-To: <5bef28df0712140707p2dd97b24se8e661b6b1c8c8c7@mail.gmail.com> References: <5bef28df0712140707p2dd97b24se8e661b6b1c8c8c7@mail.gmail.com> Message-ID: <1197928192.12657.120.camel@penguin.yourdomain.com> In addition to the recently added patches from Henrik and Alex relating to postmodern, I have added the two patches provided by Sean to the main repository. (I personally don't have LISP works, but at least these patches did not seem to break anything.) Thanks very much to Sean, Henrik and Alex for their improvements!!! On Fri, 2007-12-14 at 15:07 +0000, Sean Ross wrote: > Hi all, > here's a small patch to swap the load order of classindex-utils.lisp > and pset.lisp. > This is required as Lispworks attempts to call set-db-synch when > loading the pset fasl > which causes it to fall over. > > cheers, > sean > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Wed Dec 19 09:50:24 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 19 Dec 2007 10:50:24 +0100 (CET) Subject: [elephant-devel] Change of slot definition Message-ID: <61225.88.73.245.230.1198057824.squirrel@mail.stardawn.org> The user's guide talks about added and discarded slots, but what's the behaviour when a slot changes? Suppose I represent a picture with an object PIC that contains the location of a file holding it. I store a bunch of other objects that each have a picture: (defpclass myobj () ((picture :type PIC))) Now I decide later on that a string would be enough (defpclass myobj () ((picture :type STRING))) What will happen? A bad thing, i.e. a deserialization error? Is there a way to define how a PIC should be converted to a STRING? Leslie -- My personal blog: http://blog.viridian-project.de/ From read at robertlread.net Fri Dec 21 03:38:16 2007 From: read at robertlread.net (Robert L. Read) Date: Thu, 20 Dec 2007 21:38:16 -0600 Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <61225.88.73.245.230.1198057824.squirrel@mail.stardawn.org> References: <61225.88.73.245.230.1198057824.squirrel@mail.stardawn.org> Message-ID: <1198208296.3734.77.camel@penguin.yourdomain.com> I haven't answered this because I've spent the last three days trying to get Sean Ross's patch to work; the tests run green with it, but if you run the tests again they fail. The easiest way to answer this problem is to try it. I may have to back out the changes that Sean sent in; they interact very badly with the postmodern interface --- or at least the most recent version of it. I suspect that I may also be dealing with an SBCL bug of some kind in the MOP. I never set out to spend three days on this---the solution always seemed to be right around the corner. On Wed, 2007-12-19 at 10:50 +0100, Leslie P. Polzer wrote: > The user's guide talks about added and discarded slots, but what's the behaviour > when a slot changes? > > Suppose I represent a picture with an object PIC that contains the location of > a file holding it. I store a bunch of other objects that each have a picture: > > (defpclass myobj () > ((picture :type PIC))) > > Now I decide later on that a string would be enough > > (defpclass myobj () > ((picture :type STRING))) > > What will happen? A bad thing, i.e. a deserialization error? > Is there a way to define how a PIC should be converted to a STRING? > > Leslie > From eslick at csail.mit.edu Fri Dec 21 04:34:13 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Thu, 20 Dec 2007 23:34:13 -0500 Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <1198208296.3734.77.camel@penguin.yourdomain.com> References: <61225.88.73.245.230.1198057824.squirrel@mail.stardawn.org> <1198208296.3734.77.camel@penguin.yourdomain.com> Message-ID: Hi Leslie, Elephant does not pay any attention to type information defined as part of a slot metaobject. The lisp will complain during deserialization when you try to write the PIC to the changed slot, now of type STRING. Only OpenMCL, as far as I know, enforces type checking by default and asserts errors. Allegro and SBCL will quietly assign the wrong type to the slot, if I recall correctly. Lispworks will probably complain too. The right user approach is to provide a function to walk over instances and update their representation or zero them out. Other than an assertion on deserialization or a general warning on type change, I can't think of a really useful default policy. Have you tried this and seen what happens? What Lisp are you using? Ian On Dec 20, 2007, at 10:38 PM, Robert L. Read wrote: > I haven't answered this because I've spent the last three days > trying to get Sean Ross's patch to work; the tests run green with it, > but if you run the tests again they fail. > > The easiest way to answer this problem is to try it. > > I may have to back out the changes that Sean sent in; they interact > very > badly with the postmodern interface --- or at least the most recent > version of it. > > I suspect that I may also be dealing with an SBCL bug of some kind in > the MOP. > > I never set out to spend three days on this---the solution always > seemed > to be right around the corner. > > > > On Wed, 2007-12-19 at 10:50 +0100, Leslie P. Polzer wrote: >> The user's guide talks about added and discarded slots, but what's >> the behaviour >> when a slot changes? >> >> Suppose I represent a picture with an object PIC that contains the >> location of >> a file holding it. I store a bunch of other objects that each have >> a picture: >> >> (defpclass myobj () >> ((picture :type PIC))) >> >> Now I decide later on that a string would be enough >> >> (defpclass myobj () >> ((picture :type STRING))) >> >> What will happen? A bad thing, i.e. a deserialization error? >> Is there a way to define how a PIC should be converted to a STRING? >> >> Leslie >> > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Fri Dec 21 10:11:51 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 21 Dec 2007 11:11:51 +0100 (CET) Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <1198208296.3734.77.camel@penguin.yourdomain.com> References: <61225.88.73.245.230.1198057824.squirrel@mail.stardawn.org> <1198208296.3734.77.camel@penguin.yourdomain.com> Message-ID: <61140.88.73.193.78.1198231911.squirrel@mail.stardawn.org> I have read Sean's thread about instance initialization when it was on the list, but when it came to my special problem it didn't occur to me that it would be solved by Sean's patch. Do you think this is the case, Robert? Sean's new methods recreate-instance-using-class and recreate-instance might indeed solve my problem... although I can't pretend right now to know how they need to be used in my case. Will check this out later. Leslie -- My personal blog: http://blog.viridian-project.de/ From rosssd at gmail.com Fri Dec 21 12:48:31 2007 From: rosssd at gmail.com (Sean Ross) Date: Fri, 21 Dec 2007 12:48:31 +0000 Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <1198208296.3734.77.camel@penguin.yourdomain.com> References: <61225.88.73.245.230.1198057824.squirrel@mail.stardawn.org> <1198208296.3734.77.camel@penguin.yourdomain.com> Message-ID: <5bef28df0712210448o506cd85bmb3400eca79e924f0@mail.gmail.com> rats, Unfortunately I use the BDB backend and had not tested this with postmodern. After quick investigation my guess is that this has had adverse effects regarding objects which have specialized intialize-instance methods which are required to run an deserialization. Unfortunately I'm leaving tonight and will be away from a computer for the next 2 weeks so I'll pick this up again when I return. sorry for the fuss. Sean. On Dec 21, 2007 3:38 AM, Robert L. Read wrote: > I haven't answered this because I've spent the last three days > trying to get Sean Ross's patch to work; the tests run green with it, > but if you run the tests again they fail. > I may have to back out the changes that Sean sent in; they interact very > badly with the postmodern interface --- or at least the most recent > version of it. > > I suspect that I may also be dealing with an SBCL bug of some kind in > the MOP. > > I never set out to spend three days on this---the solution always seemed > to be right around the corner. > > > > On Wed, 2007-12-19 at 10:50 +0100, Leslie P. Polzer wrote: > > The user's guide talks about added and discarded slots, but what's the behaviour > > when a slot changes? > > > > Suppose I represent a picture with an object PIC that contains the location of > > a file holding it. I store a bunch of other objects that each have a picture: > > > > (defpclass myobj () > > ((picture :type PIC))) > > > > Now I decide later on that a string would be enough > > > > (defpclass myobj () > > ((picture :type STRING))) > > > > What will happen? A bad thing, i.e. a deserialization error? > > Is there a way to define how a PIC should be converted to a STRING? > > > > Leslie > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From leslie.polzer at gmx.net Fri Dec 21 13:03:50 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 21 Dec 2007 14:03:50 +0100 (CET) Subject: [elephant-devel] Status of Leslie's question and Sean's patch? Message-ID: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> Hi Ian, thanks for making it clear. Of course, I guessed this and the Elephant code I looked at didn't say otherwise, but you're knowledge is obviously greater. > The right user approach is to provide a function to walk over > instances and update their representation or zero them out. Other > than an assertion on deserialization or a general warning on type > change, I can't think of a really useful default policy. I can, sort of: Elephant should offer an easily usable facility to specify what should happen when the types don't match, something like (defmethod elephant:convert-slot ((from pic) (to string))) which is pretty much exactly what convert-class is for, so we might just use this. Wouldn't it suffice to just do START) type changed? [this decision should be based on the :type slot-initarg] Yes: call convert-class No: simple assignment > Have you tried this and seen what happens? Neither warning nor error, just assignment of the old data. > What Lisp are you using? I'm using SBCL/Linux/x86. This functionality is important IMHO; without it, sensible Lisp-style Rapid Prototyping is hardly possible when one needs to call map-root everytime something changes. Leslie -- My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Fri Dec 21 13:04:07 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 21 Dec 2007 14:04:07 +0100 (CET) Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: References: <61225.88.73.245.230.1198057824.squirrel@mail.stardawn.org> <1198208296.3734.77.camel@penguin.yourdomain.com> Message-ID: <63536.88.73.193.78.1198242247.squirrel@mail.stardawn.org> Hi Ian, thanks for making it clear. Of course, I guessed this and the Elephant code I looked at didn't say otherwise, but you're knowledge is obviously greater. > The right user approach is to provide a function to walk over > instances and update their representation or zero them out. Other > than an assertion on deserialization or a general warning on type > change, I can't think of a really useful default policy. I can, sort of: Elephant should offer an easily usable facility to specify what should happen when the types don't match, something like (defmethod elephant:convert-slot ((from pic) (to string))) which is pretty much exactly what convert-class is for, so we might just use this. Wouldn't it suffice to just do START) type changed? [this decision should be based on the :type slot-initarg] Yes: call convert-class No: simple assignment > Have you tried this and seen what happens? Neither warning nor error, just assignment of the old data. > What Lisp are you using? I'm using SBCL/Linux/x86. This functionality is important IMHO; without it, sensible Lisp-style Rapid Prototyping is hardly possible when one needs to call map-root everytime something changes. Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at csail.mit.edu Fri Dec 21 14:36:34 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 21 Dec 2007 09:36:34 -0500 Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> References: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> Message-ID: <8611E5B1-7828-406C-BD16-CBCF2CEB7267@csail.mit.edu> Leslie, Please correct me if I'm wrong, but isn't this exactly what Lisp does? If you assign the wrong type to a slot under SBCL, it doesn't complain so in this sense Elephant behaves in the same way as Lisp does with in-memory objects. At some point it is our responsibility not to shoot ourselves in the foot! The only reason I'm resistant is having spent some time looking at related problems. There is a big discussion in the list archives (mostly one sided from me) of ways to handle schema evolution, the more general version of the problem you point out with slot type mismatches. I would like to improve the default behavior and error handling of these cases, but the interaction between MOP functions such as update-instance-for-redefined-class method and persistent instances is pretty complicated. For example, if you don't map all instances of an object after class redefinition, then some objects are in-line with an old schema, and some with the new. That's not a problem if the update method is called on class initialization, but there is no guarantee from lisp that this will happen because you could exit and restart before you've loaded every instance. Even if you don't exist, what if you redefine the class yet again? Then there are serialized objects in the store that were written under three different schemas and no way to ensure you can update the two old ones to the new one correctly as you only have one update-instance- for-redefined-class method for a given class. If it isn't called on every instance after every redefinition, then it can result in inconsistencies. The only way to ensure this doesn't happen that I can think of is to make sure that every instance effected by the redefinition is properly updated at redefinition time. Lazy redefinition can't be guaranteed to be consistent. I'm very open to suggestions (although I'm unlikely to implement this feature anytime soon). There has been alot of discussion around this area (see Pierre and I discussing class deserialization and initialization, part of which should be solved by Sean's patch) that the initialization and reinitialization on deserialization protocol should probably be rethought. Cheers, Ian PS - Anyone know how lisp handles multiple redefinitions under the usual lazy update-on-read policy? On Dec 21, 2007, at 8:03 AM, Leslie P. Polzer wrote: > > Hi Ian, > > thanks for making it clear. Of course, I guessed this and the > Elephant code > I looked at didn't say otherwise, but you're knowledge is obviously > greater. > >> The right user approach is to provide a function to walk over >> instances and update their representation or zero them out. Other >> than an assertion on deserialization or a general warning on type >> change, I can't think of a really useful default policy. > > I can, sort of: Elephant should offer an easily usable facility > to specify what should happen when the types don't match, something > like > > (defmethod elephant:convert-slot ((from pic) (to string))) > > which is pretty much exactly what convert-class is for, so we might > just use this. > Wouldn't it suffice to just do > > START) type changed? [this decision should be based on the :type > slot-initarg] > Yes: call convert-class > No: simple assignment > > >> Have you tried this and seen what happens? > > Neither warning nor error, just assignment of the old data. > > >> What Lisp are you using? > > I'm using SBCL/Linux/x86. > > This functionality is important IMHO; without it, sensible Lisp- > style Rapid Prototyping > is hardly possible when one needs to call map-root everytime > something changes. > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Fri Dec 21 17:25:43 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 21 Dec 2007 18:25:43 +0100 (CET) Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> References: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> Message-ID: <61004.88.73.193.78.1198257943.squirrel@mail.stardawn.org> > which is pretty much exactly what convert-class is for, so we might just use this. Yeah now, that should read "change-class" of course. Sorry. Leslie -- My personal blog: http://blog.viridian-project.de/ From leslie.polzer at gmx.net Fri Dec 21 17:53:32 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 21 Dec 2007 18:53:32 +0100 (CET) Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <8611E5B1-7828-406C-BD16-CBCF2CEB7267@csail.mit.edu> References: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> <8611E5B1-7828-406C-BD16-CBCF2CEB7267@csail.mit.edu> Message-ID: <63811.88.73.193.78.1198259612.squirrel@mail.stardawn.org> > Please correct me if I'm wrong, but isn't this exactly what Lisp > does? If you assign the wrong type to a slot under SBCL, it doesn't > complain so in this sense Elephant behaves in the same way as Lisp > does with in-memory objects. At some point it is our responsibility > not to shoot ourselves in the foot! Sure, but I'm not talking about preventing that automatically, but about giving the programmer an opportunity to do agile development by specifying what should be done when data types have changed. I'm so after this not only because it'd be very useful for me, but because this is exactly the way the rest of Lisp and esp. CLOS/MOP works: if ?do what I mean? won't work, give the programmer an opportunity to clarify. > I would like to improve the default behavior and error > handling of these cases, but the interaction between MOP functions > such as update-instance-for-redefined-class method and persistent > instances is pretty complicated. For example, if you don't map all > instances of an object after class redefinition, then some objects are > in-line with an old schema, and some with the new. That's not a > problem if the update method is called on class initialization, but > there is no guarantee from lisp that this will happen because you > could exit and restart before you've loaded every instance. Not really a problem. I haven't looked at how the MOP plays into this, but the following will work: 1) slot of type A 2) slot of type B; data store partly updated 3) slot of type C as long as *both* the transitions A->C and B->C are defined. Of course, this way one has to provide a quadratic number of transitions, but: first, we assume a schema doesn't change *that* often that this becomes a problem; and if it does, the programmer could still employ map-root at some point to update them all to the latest schema. Second, we could at some point in the future provide a protocol that lets us record schema changes so we can do just A->B->C whereby a linear number of transitions is achieved. > Lazy redefinition can't be guaranteed to be consistent. With the above and the assumption that the programmer doesn't shoot herself in the foot, it can be guaranteed (correct me if I'm wrong). > I'm very open to suggestions (although I'm unlikely to implement this > feature anytime soon). There has been alot of discussion around this > area (see Pierre and I discussing class deserialization and > initialization, part of which should be solved by Sean's patch) that > the initialization and reinitialization on deserialization protocol > should probably be rethought. I will take a look at Sean's patch and the discussions you mentioned, and try to hack up some more if I see the need for it. Thanks, Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at csail.mit.edu Fri Dec 21 18:34:07 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 21 Dec 2007 13:34:07 -0500 Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <63811.88.73.193.78.1198259612.squirrel@mail.stardawn.org> References: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> <8611E5B1-7828-406C-BD16-CBCF2CEB7267@csail.mit.edu> <63811.88.73.193.78.1198259612.squirrel@mail.stardawn.org> Message-ID: On Dec 21, 2007, at 12:53 PM, Leslie P. Polzer wrote: > >> Please correct me if I'm wrong, but isn't this exactly what Lisp >> does? If you assign the wrong type to a slot under SBCL, it doesn't >> complain so in this sense Elephant behaves in the same way as Lisp >> does with in-memory objects. At some point it is our responsibility >> not to shoot ourselves in the foot! > > Sure, but I'm not talking about preventing that automatically, but > about > giving the programmer an opportunity to do agile development by > specifying what should be done when data types have changed. > > I'm so after this not only because it'd be very useful for me, but > because > this is exactly the way the rest of Lisp and esp. CLOS/MOP works: > if ?do what I mean? won't work, give the programmer an opportunity > to clarify. > >> I would like to improve the default behavior and error >> handling of these cases, but the interaction between MOP functions >> such as update-instance-for-redefined-class method and persistent >> instances is pretty complicated. For example, if you don't map all >> instances of an object after class redefinition, then some objects >> are >> in-line with an old schema, and some with the new. That's not a >> problem if the update method is called on class initialization, but >> there is no guarantee from lisp that this will happen because you >> could exit and restart before you've loaded every instance. > > Not really a problem. I haven't looked at how the MOP plays into > this, but the following will work: > > 1) slot of type A > 2) slot of type B; data store partly updated > 3) slot of type C > > as long as *both* the transitions A->C and B->C are defined. > Of course, this way one has to provide a quadratic number of > transitions, > but: first, we assume a schema doesn't change *that* often that this > becomes a problem; and if it does, the programmer could still employ > map-root at some point to update them all to the latest schema. > Second, we could at some point in the future provide a protocol > that lets us record schema changes so we can do just A->B->C > whereby a linear number of transitions is achieved. You are quite right. I should probably have said that the benefits of making this sort of thing work cleanly better has yet to justify the cost to support in my mind. Schemas and schema evolution solve quite a few problems (evolution of non-persistent objects, etc), but fitting that into the MOP and the existing Elephant is a significant piece of work touching most of the Elephant code base in one way or another. > >> Lazy redefinition can't be guaranteed to be consistent. > > With the above and the assumption that the programmer doesn't shoot > herself in the foot, it can be guaranteed (correct me if I'm wrong). It probably can be guaranteed, but the assumption set necessary to ensure it is critical. We can talk about this after you've looked at the problem some more. For example. In order to convert an instance of A to an instance of B we have to have a persistent function. The only way to store functions today are to store forms that we can compile into functions. These forms will require external dependencies to be maintained. If someone is working in a live Agile system and an A->B form is stored somewhere and a function it depends on gets redefined, it is not necessarily obvious that A->B is now performing a new function A->B'. If an archived object is read from the DB and now must go through A->B->C->D, You compound the A->B' error with B->C and C->D you can end up with a D' that is incorrect, with no knowledge why it happened because it was buried in a function you wrote, assumed was applied and never looked at again. Perhaps rather than an implicit feature, we provide a toolkit for making user schemas easy to build. This could make it easy to create, compare and perform operations on schemas from classes and to hook into the MOP to extract them and use them in class redefinition and creation. We could provide an inherited slot for class instances that keeps a pointer to the schema. Users that are afraid of this functionality can just choose to update manually so that consistence is guaranteed. Trading off performance for a strong guarantee strikes me as the better default behavior. > >> I'm very open to suggestions (although I'm unlikely to implement this >> feature anytime soon). There has been alot of discussion around this >> area (see Pierre and I discussing class deserialization and >> initialization, part of which should be solved by Sean's patch) that >> the initialization and reinitialization on deserialization protocol >> should probably be rethought. > > I will take a look at Sean's patch and the discussions you mentioned, > and try to hack up some more if I see the need for it. You might want to look through the Trac database and Wiki where all of the existing proposals for improving Elephant are documented. I believe schema evolution is on the roadmap for 0.9.x, but I'm only bug fixing these days and don't have the bandwidth to take on the major projects I had lined up for 0.9.x before I got pulled away. I'm happy to support this effort if you or someone else are interested in taking it on, though. > Thanks, > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Fri Dec 21 19:12:09 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Fri, 21 Dec 2007 20:12:09 +0100 (CET) Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: References: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> <8611E5B1-7828-406C-BD16-CBCF2CEB7267@csail.mit.edu> <63811.88.73.193.78.1198259612.squirrel@mail.stardawn.org> Message-ID: <61759.88.73.193.78.1198264329.squirrel@mail.stardawn.org> > For example. In order to convert an instance of A to an instance of B > we have to have a persistent function. Let's stop right here. Why that? What I had in mind was that the programmer keeps around, say, a file full of those conversion functions, thereby providing legacy support for her data structures. > Perhaps rather than an implicit feature, we provide a toolkit for > making user schemas easy to build. This could make it easy to > create, compare and perform operations on schemas from classes and to > hook into the MOP to extract them and use them in class redefinition > and creation. We could provide an inherited slot for class instances > that keeps a pointer to the schema. Users that are afraid of this > functionality can just choose to update manually so that consistence > is guaranteed. Trading off performance for a strong guarantee strikes > me as the better default behavior. I don't understand what you have in mind here. Perhaps it would help if you explained the idea of a ?user schema? in more detail. Then again, I don't think this is a thing that is of interest to me right now, so you might not want to explain this further and save your time. > I'm happy to support this effort if you or someone else are interested > in taking it on, though. Thanks a lot. I'll do what I can. I must balance between the needs of my project and Elephant, though. Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at csail.mit.edu Sat Dec 22 00:19:32 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Fri, 21 Dec 2007 19:19:32 -0500 Subject: [elephant-devel] Status of Leslie's question and Sean's patch? In-Reply-To: <61759.88.73.193.78.1198264329.squirrel@mail.stardawn.org> References: <63504.88.73.193.78.1198242230.squirrel@mail.stardawn.org> <8611E5B1-7828-406C-BD16-CBCF2CEB7267@csail.mit.edu> <63811.88.73.193.78.1198259612.squirrel@mail.stardawn.org> <61759.88.73.193.78.1198264329.squirrel@mail.stardawn.org> Message-ID: <6B64EE1C-46D7-45EC-ACCE-41D10C423B13@csail.mit.edu> From what I can tell you are proposing that we make a hook into the instance creation process to clean up these redefinition and change- class issues. It sounds like you aren't concerned with fitting within the existing MOP framework (a source of much of my concern). The 'toolkit' proposal is essentially this; create a mechanism outside (but not conflicting with) the MOP that solves these problems cleanly. A few questions to consider: - What code is going to generate and keep track of the schema number for any persistent instance? - Where are schema 'numbers' going to be stored? - What happens if a persistent class is redefined, but the resulting database is not connected or is never connected (so you can't persist the information that this class was redefined)? - How does the system know which of these conversion functions to call during instantiation? (What code is mapping schema 'numbers' to functions, whether stored in the DB or stored in a file?) - When is the conversion function called? When the pid is read into memory and a placeholder object created, or when the first slot value is accessed? - How does this mechanism interact with user-defined MOP conversion functions such as update-instance-for-redefined-class? Are they the same interface or separate? - What are the potential error cases we should warn the user about? I wish we had a developer who was passionate about this area of the code in general. There are a set of important open issues that all effect the same mechanisms (numbers refer to Trac tickets on http://trac.common-lisp.net/elephant/) 1) Schema evolution for persistent instances and non-persistent objects (#25 - Leslie) 2) Lazy object instantiation for persistent instances; clean up instance initialization to support both 'on creation' and 'on deserialization' pathways. (#39/49 - Pierre & Sean) 3) Referential integrity (#3 - Ian) 4) Online garbage collection (#21) and migration (#46) 5) 64-bit OIDs (#29) As before, I'm happy to support but probably can't take the lead on this for a very long time. Ian On Dec 21, 2007, at 2:12 PM, Leslie P. Polzer wrote: > >> For example. In order to convert an instance of A to an instance >> of B >> we have to have a persistent function. > > Let's stop right here. Why that? > > What I had in mind was that the programmer keeps around, say, a file > full of those conversion functions, thereby providing legacy support > for her data structures. > > >> Perhaps rather than an implicit feature, we provide a toolkit for >> making user schemas easy to build. This could make it easy to >> create, compare and perform operations on schemas from classes and to >> hook into the MOP to extract them and use them in class redefinition >> and creation. We could provide an inherited slot for class instances >> that keeps a pointer to the schema. Users that are afraid of this >> functionality can just choose to update manually so that consistence >> is guaranteed. Trading off performance for a strong guarantee >> strikes >> me as the better default behavior. > > I don't understand what you have in mind here. Perhaps it would help > if you explained the idea of a ?user schema? in more detail. > Then again, I don't think this is a thing that is of interest to me > right now, so you might not want to explain this further and save > your time. > > >> I'm happy to support this effort if you or someone else are >> interested >> in taking it on, though. > > Thanks a lot. I'll do what I can. I must balance between the needs of > my project and Elephant, though. > > Leslie > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Wed Dec 26 10:50:45 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 26 Dec 2007 11:50:45 +0100 (CET) Subject: [elephant-devel] classes-new.lisp / classes.lisp Message-ID: <59465.84.157.17.32.1198666245.squirrel@mail.stardawn.org> What's up with those two? Does the first make the latter obsolete? Leslie From eslick at csail.mit.edu Wed Dec 26 13:25:46 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 26 Dec 2007 08:25:46 -0500 Subject: [elephant-devel] classes-new.lisp / classes.lisp In-Reply-To: <59465.84.157.17.32.1198666245.squirrel@mail.stardawn.org> References: <59465.84.157.17.32.1198666245.squirrel@mail.stardawn.org> Message-ID: I believe classes-new contains some in-progress edits I was working on (converting to Pascal's MOP compatibility package). If you check the elephant.asd file, whichever file is loaded is the active ones and the others should be ignored, but not deleted! Ian On Dec 26, 2007, at 5:50 AM, Leslie P. Polzer wrote: > > What's up with those two? > Does the first make the latter obsolete? > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Wed Dec 26 16:07:13 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 26 Dec 2007 17:07:13 +0100 (CET) Subject: [elephant-devel] Wrong docstring in classes.lisp? Message-ID: <59947.84.157.22.183.1198685233.squirrel@mail.stardawn.org> (defmethod change-class :around ((previous standard-object) (new-class persistent-metaclass) &rest initargs) (declare (ignorable initargs)) (unless (subtypep (type-of previous) 'persistent) (error "Standard classes cannot be changed to non-persistent classes in change-class")) (call-next-method)) Isn't this supposed to read "persistent" instead of "non-persistent"? -- My personal blog: http://blog.viridian-project.de/ From eslick at csail.mit.edu Wed Dec 26 16:25:18 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 26 Dec 2007 11:25:18 -0500 Subject: [elephant-devel] Wrong docstring in classes.lisp? In-Reply-To: <59947.84.157.22.183.1198685233.squirrel@mail.stardawn.org> References: <59947.84.157.22.183.1198685233.squirrel@mail.stardawn.org> Message-ID: <14504F10-37BC-4D75-9C2F-487747022943@csail.mit.edu> That is correct. I'll patch that. -Ian On Dec 26, 2007, at 11:07 AM, Leslie P. Polzer wrote: > > (defmethod change-class :around ((previous standard-object) (new-class > persistent-metaclass) &rest initargs) > (declare (ignorable initargs)) > (unless (subtypep (type-of previous) 'persistent) > (error "Standard classes cannot be changed to non-persistent > classes in change-class")) > (call-next-method)) > > Isn't this supposed to read "persistent" instead of "non-persistent"? > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Wed Dec 26 16:39:53 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 26 Dec 2007 17:39:53 +0100 (CET) Subject: [elephant-devel] Untangling "serialize" Message-ID: <64663.84.157.22.183.1198687193.squirrel@mail.stardawn.org> It's not really of immediate concern for me, but wouldn't it probably make sense to split the "serialize" function into a generic function and a bunch of methods? It's currently a giant typecase mudball. It would in any case be more extensible this way. Leslie -- My personal blog: http://blog.viridian-project.de/ From eslick at csail.mit.edu Wed Dec 26 19:42:19 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 26 Dec 2007 14:42:19 -0500 Subject: [elephant-devel] Untangling "serialize" In-Reply-To: <64663.84.157.22.183.1198687193.squirrel@mail.stardawn.org> References: <64663.84.157.22.183.1198687193.squirrel@mail.stardawn.org> Message-ID: <9BAAAF3C-742F-420C-9D5D-3CE3124532FE@csail.mit.edu> Yes and no. I think the tradeoffs are not totally obvious. The serializer/deserializer was written by the original authors for performance reasons. Arrays, hashtables and lists in particular require lots of recursive calls to %serialize/%deserialize. That can translate into many generic function dispatches in an inner loop. It's not clear how large the performance impact would be though; I haven't been motivated enough yet to test the change under enough different usage models to be sure it doesn't have a deleterious effect on any one. You'd also have to push some of the shared let bindings into global variables or generic function arguments such as the serializer-hash and id counter. I did something similar to this with the migrate function (migrate.lisp), but there the performance didn't matter. I'm not sure that code is any cleaner for being broken up than it would be if I had put it all in one function with typecase lines to clearly delineate type-specific functionality. It would be interesting to see a performance comparison between the optimized and general approach and whether the other formulation reads better. Ian On Dec 26, 2007, at 11:39 AM, Leslie P. Polzer wrote: > > It's not really of immediate concern for me, but wouldn't it > probably make sense to split the "serialize" function into > a generic function and a bunch of methods? It's currently > a giant typecase mudball. > > It would in any case be more extensible this way. > > Leslie > > > -- > My personal blog: http://blog.viridian-project.de/ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From midfield at gmail.com Thu Dec 27 04:01:23 2007 From: midfield at gmail.com (midfield at gmail.com) Date: Thu, 27 Dec 2007 09:31:23 +0530 Subject: [elephant-devel] Untangling "serialize" In-Reply-To: <9BAAAF3C-742F-420C-9D5D-3CE3124532FE@csail.mit.edu> References: <64663.84.157.22.183.1198687193.squirrel@mail.stardawn.org> <9BAAAF3C-742F-420C-9D5D-3CE3124532FE@csail.mit.edu> Message-ID: <9157df230712262001j71165184wafc57719efa4a77@mail.gmail.com> i was trying to be smarter than the compiler -- i don't remember if PCL can do runtime profiling directed dispatch tables -- i attempted to generate the optimal dispatch table "by hand." as moon would say, given a sufficiently smart compiler.....the generic function version would be better. probably ACL and Lispworks are ahead in this way than the PCL-based lisps. b On 12/27/07, Ian Eslick wrote: > Yes and no. I think the tradeoffs are not totally obvious. > > The serializer/deserializer was written by the original authors for > performance reasons. Arrays, hashtables and lists in particular > require lots of recursive calls to %serialize/%deserialize. That can > translate into many generic function dispatches in an inner loop. > It's not clear how large the performance impact would be though; I > haven't been motivated enough yet to test the change under enough > different usage models to be sure it doesn't have a deleterious effect > on any one. > > You'd also have to push some of the shared let bindings into global > variables or generic function arguments such as the serializer-hash > and id counter. > > I did something similar to this with the migrate function > (migrate.lisp), but there the performance didn't matter. I'm not sure > that code is any cleaner for being broken up than it would be if I had > put it all in one function with typecase lines to clearly delineate > type-specific functionality. > > It would be interesting to see a performance comparison between the > optimized and general approach and whether the other formulation reads > better. > > Ian > > > On Dec 26, 2007, at 11:39 AM, Leslie P. Polzer wrote: > > > > > It's not really of immediate concern for me, but wouldn't it > > probably make sense to split the "serialize" function into > > a generic function and a bunch of methods? It's currently > > a giant typecase mudball. > > > > It would in any case be more extensible this way. > > > > Leslie > > > > > > -- > > My personal blog: http://blog.viridian-project.de/ > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From evrim at core.gen.tr Thu Dec 27 09:49:46 2007 From: evrim at core.gen.tr (Evrim Ulu) Date: Thu, 27 Dec 2007 11:49:46 +0200 Subject: [elephant-devel] Untangling "serialize" In-Reply-To: <9157df230712262001j71165184wafc57719efa4a77@mail.gmail.com> References: <64663.84.157.22.183.1198687193.squirrel@mail.stardawn.org> <9BAAAF3C-742F-420C-9D5D-3CE3124532FE@csail.mit.edu> <9157df230712262001j71165184wafc57719efa4a77@mail.gmail.com> Message-ID: <4773753A.9080602@core.gen.tr> midfield at gmail.com wrote: > i was trying to be smarter than the compiler -- i don't remember if > PCL can do runtime profiling directed dispatch tables -- i attempted > to generate the optimal dispatch table "by hand." as moon would say, > given a sufficiently smart compiler.....the generic function version > would be better. probably ACL and Lispworks are ahead in this way > than the PCL-based lisps. > SBCL generic function dispatcher is faster than function + typecase. evrim. From leslie.polzer at gmx.net Thu Dec 27 11:06:58 2007 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Thu, 27 Dec 2007 12:06:58 +0100 (CET) Subject: [elephant-devel] Quick sketch of type evolution system Message-ID: <62451.84.157.22.183.1198753618.squirrel@mail.stardawn.org> Hello everyone, to clarify one of the points that are important to me, I have prepared this working sketch: (defmethod slot-value-using-class ((class persistent-metaclass) (instance persistent-object) (slot-def persistent-slot-definition)) "Get the slot value from the database." (let* ((name (slot-definition-name slot-def)) (value (persistent-slot-reader (get-con instance) instance name)) (data-type (type-of value)) ; type of data in store (slot-type (sb-mop::slot-definition-type slot-def))) ; type declared in defclass (format t "slot-value-using-class for slot ~A (declared type '~A'); value '~A', type '~A'~%" name slot-type value data-type) (if (typep data-type slot-type) value #| here we would call the user-provided change-class for the old and new types |# ))) Notes: * it's obviously implementation dependent as of yet, but this is trivial. * we should provide a way for the user to specify that this type checking shouldn't take place, for performance reasons. This is simple, too. * as you can see, I don't care much about evolution at the class level (class renamed, substituted, ...) right now, but at the slot level. * I don't see any other way than slot-value-using-class to catch type changes. They can occur at every place in time between slot reads. * the replacement of slots (one slot discarded, new slot added -- how do we connect this?) could be done by a ":replaces" slot initarg. But this collides with multiple subsequent replacements, so I think the most sensible way would be a new class option named "version" of type integer. This would facilitate a user-level schema evolution protocol as outlined in the recent discussion: (defmethod elephant:class-evolve ((class myclass) (version (eql 1)) (version (eql 2))) [obviously would implement transition 1->2 for class "foo"] And Ian, thanks for your concise, patient and quick answers. I thought one could never get *all* three of those :) Leslie -- My personal blog: http://blog.viridian-project.de/ From rosssd at gmail.com Fri Dec 28 18:28:14 2007 From: rosssd at gmail.com (Sean Ross) Date: Fri, 28 Dec 2007 19:28:14 +0100 Subject: [elephant-devel] Untangling "serialize" In-Reply-To: <4773753A.9080602@core.gen.tr> References: <64663.84.157.22.183.1198687193.squirrel@mail.stardawn.org> <9BAAAF3C-742F-420C-9D5D-3CE3124532FE@csail.mit.edu> <9157df230712262001j71165184wafc57719efa4a77@mail.gmail.com> <4773753A.9080602@core.gen.tr> Message-ID: <5bef28df0712281028v4bc67f4cr654e5a43662a1282@mail.gmail.com> AFAIK this goes for SBCL, CMUCL, Lispworks and ACL. On 12/27/07, Evrim Ulu wrote: > SBCL generic function dispatcher is faster than function + typecase. > > evrim. > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From varuzza at gmail.com Sun Dec 30 00:24:14 2007 From: varuzza at gmail.com (Leonardo Varuzza) Date: Sat, 29 Dec 2007 22:24:14 -0200 Subject: [elephant-devel] Fix the use of internal symbol of sb-kernel in memutils Message-ID: <5d9c2e640712291624t5e26ffc0m45fd4bc249f764cf@mail.gmail.com> Hello, I found a minor problem compiling memutil.lisp , because the functions sb-kernel::copy-*-from-system-area aren't exported in the latest version of sbcl. I made a little patch to fix it. Best Regards, Leonardo Varuzza. -- The most fundamental particles in this product are held together by a "gluing" force about which little is currently known and whose adhesive power can therefore not be permanently guaranteed. -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-the-use-of-internal-symbol-of-sb_kernel-in-memutils.dpatch Type: text/x-darcs-patch Size: 925 bytes Desc: not available URL: From kazennikov.nntp at gmail.com Sun Dec 30 14:59:07 2007 From: kazennikov.nntp at gmail.com (Anton Kazennikov) Date: Sun, 30 Dec 2007 17:59:07 +0300 Subject: [elephant-devel] bugfix for db-bdb: utf16 string comparison Message-ID: <87ir2gdy9w.fsf@kzn.homelinux.org> Hello all! I found a bug in db-bdb string comparison functions - when comparing two utf16 strings only the first half of strings are compared. (bug is in misinterpreting string size in chars and in bytes). -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: bdb-bugfix.txt URL: -------------- next part -------------- -- With best regards, Anton Kazennikov. mailto:kazennikov[at]mirea.ru ICQ# 98965967 From kazennikov.nntp at gmail.com Sun Dec 30 15:48:35 2007 From: kazennikov.nntp at gmail.com (Anton Kazennikov) Date: Sun, 30 Dec 2007 18:48:35 +0300 Subject: [elephant-devel] patch: update db-bdb to Berkeley DB 4.6 Message-ID: <87ejd4dvzg.fsf@kzn.homelinux.org> Hello! I've updated db-bdb backend to Berkeley DB 4.6. The main problem is that the API constants during 4.5 -> 4.6 have changed. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: db46.txt URL: -------------- next part -------------- -- With best regards, Anton Kazennikov. mailto:kazennikov[at]mirea.ru ICQ# 98965967 From eslick at csail.mit.edu Sun Dec 30 17:06:52 2007 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 30 Dec 2007 12:06:52 -0500 Subject: [elephant-devel] patch: update db-bdb to Berkeley DB 4.6 In-Reply-To: <87ejd4dvzg.fsf@kzn.homelinux.org> References: <87ejd4dvzg.fsf@kzn.homelinux.org> Message-ID: Great, thank you for the patches! The update from BDB 4.x to 4.(x+1) have all had this constant problem. At some point we should figure out how to generate the lisp code that matches this file so we don't have to do this again and can link against any version... But for now we'll release this patch in contrib so people using the development branch can run 4.5 (default) or 4.6 via the patch. Ian On Dec 30, 2007, at 10:48 AM, Anton Kazennikov wrote: > Hello! > > I've updated db-bdb backend to Berkeley DB 4.6. The main problem is > that the > API constants during 4.5 -> 4.6 have changed. > New patches: > > [db-bdb updated to BerkeleyDB 4.6 > kazennikov at gmail.com**20071230140731 > Changed only BDB constants as upgrade 4.5 -> 4.6 they were > changed. > A kind of hack perhaps. But it works. The testing was not excessive, > but it works well for my project. > ] { > hunk ./src/db-bdb/berkeley-db.lisp 92 > -(defconstant DB-BTREE 1) > -(defconstant DB-HASH 2) > -(defconstant DB-RECNO 3) > -(defconstant DB-QUEUE 4) > -(defconstant DB-UNKNOWN 5) > hunk ./src/db-bdb/berkeley-db.lisp 93 > -(defconstant DB_LOCK_NOWAIT #x00000002) > +;; from cffi.lisp of cl-berkeley-db > +(cl:defconstant DB_CREATE #x0000001) > + > +(cl:defconstant DB_DURABLE_UNKNOWN #x0000002) > + > +(cl:defconstant DB_FORCE #x0000004) > + > +(cl:defconstant DB_MULTIVERSION #x0000008) > + > +(cl:defconstant DB_NOMMAP #x0000010) > + > +(cl:defconstant DB_RDONLY #x0000020) > + > +(cl:defconstant DB_RECOVER #x0000040) > + > +(cl:defconstant DB_THREAD #x0000080) > + > +(cl:defconstant DB_TRUNCATE #x0000100) > + > +(cl:defconstant DB_TXN_NOSYNC #x0000200) > + > +(cl:defconstant DB_TXN_NOWAIT #x0000400) > + > +(cl:defconstant DB_TXN_NOT_DURABLE #x0000800) > + > +(cl:defconstant DB_TXN_WRITE_NOSYNC #x0001000) > + > +(cl:defconstant DB_SPARE_FLAG #x0002000) > + > +(cl:defconstant DB_IGNORE_LEASE #x01000000) > + > +(cl:defconstant DB_AUTO_COMMIT #x02000000) > + > +(cl:defconstant DB_READ_COMMITTED #x04000000) > + > +(cl:defconstant DB_DEGREE_2 #x04000000) > + > +(cl:defconstant DB_READ_UNCOMMITTED #x08000000) > + > +(cl:defconstant DB_DIRTY_READ #x08000000) > + > +(cl:defconstant DB_TXN_SNAPSHOT #x10000000) > + > +(cl:defconstant DB_CXX_NO_EXCEPTIONS #x0000001) > + > +(cl:defconstant DB_RPCCLIENT #x0000002) > + > +(cl:defconstant DB_XA_CREATE #x0000002) > + > +(cl:defconstant DB_USE_ENVIRON #x0004000) > + > +(cl:defconstant DB_USE_ENVIRON_ROOT #x0008000) > + > +(cl:defconstant DB_INIT_CDB #x0010000) > + > +(cl:defconstant DB_INIT_LOCK #x0020000) > + > +(cl:defconstant DB_INIT_LOG #x0040000) > + > +(cl:defconstant DB_INIT_MPOOL #x0080000) > + > +(cl:defconstant DB_INIT_REP #x0100000) > + > +(cl:defconstant DB_INIT_TXN #x0200000) > + > +(cl:defconstant DB_LOCKDOWN #x0400000) > + > +(cl:defconstant DB_PRIVATE #x0800000) > + > +(cl:defconstant DB_RECOVER_FATAL #x1000000) > + > +(cl:defconstant DB_REGISTER #x2000000) > + > +(cl:defconstant DB_SYSTEM_MEM #x4000000) > + > +(cl:defconstant DB_JOINENV #x0) > + > +(cl:defconstant DB_EXCL #x0004000) > + > +(cl:defconstant DB_FCNTL_LOCKING #x0008000) > + > +(cl:defconstant DB_NO_AUTO_COMMIT #x0010000) > + > +(cl:defconstant DB_RDWRMASTER #x0020000) > + > +(cl:defconstant DB_WRITEOPEN #x0040000) > + > +(cl:defconstant DB_IMMUTABLE_KEY #x0004000) > + > +(cl:defconstant DB_TXN_SYNC #x0004000) > + > +(cl:defconstant DB_TXN_WAIT #x0008000) > + > +(cl:defconstant DB_CKP_INTERNAL #x0004000) > + > +(cl:defconstant DB_ENCRYPT_AES #x0000001) > + > +(cl:defconstant DB_CDB_ALLDB #x00004000) > + > +(cl:defconstant DB_DIRECT_DB #x00008000) > + > +(cl:defconstant DB_DIRECT_LOG #x00010000) > + > +(cl:defconstant DB_DSYNC_DB #x00020000) > + > +(cl:defconstant DB_DSYNC_LOG #x00040000) > + > +(cl:defconstant DB_LOG_AUTOREMOVE #x00080000) > + > +(cl:defconstant DB_LOG_INMEMORY #x00100000) > + > +(cl:defconstant DB_NOLOCKING #x00200000) > + > +(cl:defconstant DB_NOPANIC #x00400000) > + > +(cl:defconstant DB_OVERWRITE #x00800000) > + > +(cl:defconstant DB_PANIC_ENVIRONMENT #x01000000) > + > +(cl:defconstant DB_REGION_INIT #x20000000) > + > +(cl:defconstant DB_TIME_NOTGRANTED #x40000000) > + > +(cl:defconstant DB_YIELDCPU #x80000000) > + > +(cl:defconstant DB_UPGRADE #x0000001) > + > +(cl:defconstant DB_VERIFY #x0000002) > + > +(cl:defconstant DB_FREELIST_ONLY #x00004000) > + > +(cl:defconstant DB_FREE_SPACE #x00008000) > + > +(cl:defconstant DB_COMPACT_FLAGS (cl:logior #x00004000 #x00008000)) > + > +(cl:defconstant DB_DIRECT #x0004000) > + > +(cl:defconstant DB_EXTENT #x0008000) > + > +(cl:defconstant DB_ODDFILESIZE #x0010000) > + > +(cl:defconstant DB_CHKSUM #x00004000) > + > +(cl:defconstant DB_DUP #x00008000) > + > +(cl:defconstant DB_DUPSORT #x00010000) > + > +(cl:defconstant DB_ENCRYPT #x00020000) > + > +(cl:defconstant DB_INORDER #x00040000) > + > +(cl:defconstant DB_RECNUM #x00080000) > + > +(cl:defconstant DB_RENUMBER #x00100000) > + > +(cl:defconstant DB_REVSPLITOFF #x00200000) > + > +(cl:defconstant DB_SNAPSHOT #x00400000) > + > +(cl:defconstant DB_FAST_STAT #x0000001) > + > +(cl:defconstant DB_STAT_ALL #x0000002) > + > +(cl:defconstant DB_STAT_CLEAR #x0000004) > + > +(cl:defconstant DB_STAT_LOCK_CONF #x0000008) > + > +(cl:defconstant DB_STAT_LOCK_LOCKERS #x0000010) > + > +(cl:defconstant DB_STAT_LOCK_OBJECTS #x0000020) > + > +(cl:defconstant DB_STAT_LOCK_PARAMS #x0000040) > + > +(cl:defconstant DB_STAT_MEMP_HASH #x0000080) > + > +(cl:defconstant DB_STAT_NOERROR #x0000100) > + > +(cl:defconstant DB_STAT_SUBSYSTEM #x0000200) > + > +(cl:defconstant DB_JOIN_NOSORT #x0000001) > + > +(cl:defconstant DB_AGGRESSIVE #x0000001) > + > +(cl:defconstant DB_NOORDERCHK #x0000002) > + > +(cl:defconstant DB_ORDERCHKONLY #x0000004) > + > +(cl:defconstant DB_PR_PAGE #x0000008) > + > +(cl:defconstant DB_PR_RECOVERYTEST #x0000010) > + > +(cl:defconstant DB_PRINTABLE #x0000020) > + > +(cl:defconstant DB_SALVAGE #x0000040) > + > +(cl:defconstant DB_UNREF #x0000080) > + > +(cl:defconstant DB_REP_ANYWHERE #x0000001) > + > +(cl:defconstant DB_REP_NOBUFFER #x0000002) > + > +(cl:defconstant DB_REP_PERMANENT #x0000004) > + > +(cl:defconstant DB_REP_REREQUEST #x0000008) > + > +(cl:defconstant DB_MUTEX_ALLOCATED #x01) > + > +(cl:defconstant DB_MUTEX_LOCKED #x02) > + > +(cl:defconstant DB_MUTEX_LOGICAL_LOCK #x04) > hunk ./src/db-bdb/berkeley-db.lisp 304 > -(defconstant DB_CREATE #x00000001) > -(defconstant DB_FORCE #x00000004) > -(defconstant DB_MULTIVERSION #x00000008) > -(defconstant DB_NOMMAP #x00000010) > -(defconstant DB_RDONLY #x00000020) > -(defconstant DB_RECOVER #x00000040) > -(defconstant DB_THREAD #x00000080) > -(defconstant DB_TRUNCATE #x00000100) > -(defconstant DB_TXN_NOSYNC #x00000200) > -(defconstant DB_TXN_NOT_DURABLE #x00000400) > -(defconstant DB_TXN_WRITE_NOSYNC #x00000800) > +(cl:defconstant DB_MUTEX_PROCESS_ONLY #x08) > hunk ./src/db-bdb/berkeley-db.lisp 306 > -(defconstant DB_EXCL #x00004000) > +(cl:defconstant DB_MUTEX_SELF_BLOCK #x10) > hunk ./src/db-bdb/berkeley-db.lisp 308 > -(defconstant DB_TXN_NOWAIT #x00004000) > -(defconstant DB_TXN_SYNC #x00008000) > hunk ./src/db-bdb/berkeley-db.lisp 309 > -(defconstant DB_DUP #x00008000) > -(defconstant DB_DUPSORT #x00010000) > +(cl:defconstant DB_THREADID_STRLEN 128) > hunk ./src/db-bdb/berkeley-db.lisp 311 > -(defconstant DB_JOINENV #x00000000) > -(defconstant DB_INIT_CDB #x00004000) > -(defconstant DB_INIT_LOCK #x00008000) > -(defconstant DB_INIT_LOG #x00010000) > -(defconstant DB_INIT_MPOOL #x00020000) > -(defconstant DB_INIT_REP #x00040000) > -(defconstant DB_INIT_TXN #x00080000) > -(defconstant DB_LOCKDOWN #x00100000) > -(defconstant DB_PRIVATE #x00200000) > -(defconstant DB_RECOVER_FATAL #x00400000) > -(defconstant DB_REGISTER #x00800000) > -(defconstant DB_SYSTEM_MEM #x01000000) > -(defconstant DB_AUTO_COMMIT #x02000000) > -(defconstant DB_READ_COMMITTED #x04000000) > -(defconstant DB_DEGREE_2 #x04000000) ;; DEPRECATED, now > called DB_READ_COMMITTED > -(defconstant DB_READ_UNCOMMITTED #x08000000) > -(defconstant DB_DIRTY_READ #x08000000) ;; DEPRECATED, now > called DB_READ_UNCOMMITTED > +(cl:defconstant DB_LOCKVERSION 1) > hunk ./src/db-bdb/berkeley-db.lisp 313 > -(defconstant DB_AFTER 1) > -(defconstant DB_BEFORE 3) > -(defconstant DB_CURRENT 6) > -(defconstant DB_FIRST 7) > -(defconstant DB_GET_BOTH 8) > -(defconstant DB_GET_BOTH_RANGE 10) > -(defconstant DB_KEYFIRST 13) > -(defconstant DB_KEYLAST 14) > -(defconstant DB_LAST 15) > -(defconstant DB_NEXT 16) > -(defconstant DB_NEXT_DUP 17) > -(defconstant DB_NEXT_NODUP 18) > -(defconstant DB_PREV 23) > -(defconstant DB_PREV_NODUP 24) > -(defconstant DB_SET 25) > -(defconstant DB_SET_RANGE 27) > +(cl:defconstant DB_FILE_ID_LEN 20) > hunk ./src/db-bdb/berkeley-db.lisp 315 > -(defconstant DB_NODUPDATA 19) > -(defconstant DB_NOOVERWRITE 20) > -(defconstant DB_NOSYNC 21) > +(cl:defconstant DB_LOCK_NORUN 0) > hunk ./src/db-bdb/berkeley-db.lisp 317 > -(defconstant DB_POSITION 22) > +(cl:defconstant DB_LOCK_DEFAULT 1) > hunk ./src/db-bdb/berkeley-db.lisp 319 > -(defconstant DB_SEQ_DEC #x00000001) > -(defconstant DB_SEQ_INC #x00000002) > -(defconstant DB_SEQ_WRAP #x00000008) > +(cl:defconstant DB_LOCK_EXPIRE 2) > hunk ./src/db-bdb/berkeley-db.lisp 321 > -(defconstant DB_SET_LOCK_TIMEOUT 26) > -(defconstant DB_SET_TXN_TIMEOUT 30) > +(cl:defconstant DB_LOCK_MAXLOCKS 3) > hunk ./src/db-bdb/berkeley-db.lisp 323 > -(defconstant DB_FREELIST_ONLY #x00004000) > -(defconstant DB_FREE_SPACE #x00008000) > +(cl:defconstant DB_LOCK_MAXWRITE 4) > hunk ./src/db-bdb/berkeley-db.lisp 325 > -(defconstant DB_KEYEMPTY -30997) > -(defconstant DB_KEYEXIST -30996) > -(defconstant DB_LOCK_DEADLOCK -30995) > -(defconstant DB_LOCK_NOTGRANTED -30994) > -(defconstant DB_NOTFOUND -30989) > +(cl:defconstant DB_LOCK_MINLOCKS 5) > + > +(cl:defconstant DB_LOCK_MINWRITE 6) > + > +(cl:defconstant DB_LOCK_OLDEST 7) > + > +(cl:defconstant DB_LOCK_RANDOM 8) > + > +(cl:defconstant DB_LOCK_YOUNGEST 9) > + > +(cl:defconstant DB_LOCK_ABORT #x001) > + > +(cl:defconstant DB_LOCK_NOWAIT #x002) > + > +(cl:defconstant DB_LOCK_RECORD #x004) > + > +(cl:defconstant DB_LOCK_SET_TIMEOUT #x008) > + > +(cl:defconstant DB_LOCK_SWITCH #x010) > + > +(cl:defconstant DB_LOCK_UPGRADE #x020) > + > +(cl:defconstant DB_SET_LOCK_TIMEOUT 1) > + > +(cl:defconstant DB_SET_TXN_NOW 2) > + > +(cl:defconstant DB_SET_TXN_TIMEOUT 3) > + > +(cl:defconstant DB_RENAMEMAGIC #x030800) > + > +(cl:defconstant DB_BTREEVERSION 9) > + > +(cl:defconstant DB_BTREEOLDVER 8) > + > +(cl:defconstant DB_BTREEMAGIC #x053162) > + > +(cl:defconstant DB_HASHVERSION 9) > + > +(cl:defconstant DB_HASHOLDVER 7) > + > +(cl:defconstant DB_HASHMAGIC #x061561) > + > +(cl:defconstant DB_QAMVERSION 4) > + > +(cl:defconstant DB_QAMOLDVER 3) > + > +(cl:defconstant DB_QAMMAGIC #x042253) > + > +(cl:defconstant DB_SEQUENCE_VERSION 2) > + > +(cl:defconstant DB_SEQUENCE_OLDVER 1) > + > +(cl:defconstant DB_AFTER 1) > + > +(cl:defconstant DB_APPEND 2) > + > +(cl:defconstant DB_BEFORE 3) > + > +(cl:defconstant DB_CONSUME 4) > + > +(cl:defconstant DB_CONSUME_WAIT 5) > + > +(cl:defconstant DB_CURRENT 6) > + > +(cl:defconstant DB_FIRST 7) > + > +(cl:defconstant DB_GET_BOTH 8) > + > +(cl:defconstant DB_GET_BOTHC 9) > + > +(cl:defconstant DB_GET_BOTH_RANGE 10) > + > +(cl:defconstant DB_GET_RECNO 11) > + > +(cl:defconstant DB_JOIN_ITEM 12) > + > +(cl:defconstant DB_KEYFIRST 13) > + > +(cl:defconstant DB_KEYLAST 14) > + > +(cl:defconstant DB_LAST 15) > + > +(cl:defconstant DB_NEXT 16) > + > +(cl:defconstant DB_NEXT_DUP 17) > + > +(cl:defconstant DB_NEXT_NODUP 18) > + > +(cl:defconstant DB_NODUPDATA 19) > + > +(cl:defconstant DB_NOOVERWRITE 20) > + > +(cl:defconstant DB_NOSYNC 21) > + > +(cl:defconstant DB_POSITION 22) > + > +(cl:defconstant DB_PREV 23) > + > +(cl:defconstant DB_PREV_DUP 24) > + > +(cl:defconstant DB_PREV_NODUP 25) > + > +(cl:defconstant DB_SET 26) > + > +(cl:defconstant DB_SET_RANGE 27) > + > +(cl:defconstant DB_SET_RECNO 28) > + > +(cl:defconstant DB_UPDATE_SECONDARY 29) > + > +(cl:defconstant DB_WRITECURSOR 30) > + > +(cl:defconstant DB_WRITELOCK 31) > + > +(cl:defconstant DB_OPFLAGS_MASK #x000000ff) > + > +(cl:defconstant DB_MULTIPLE #x10000000) > + > +(cl:defconstant DB_MULTIPLE_KEY #x20000000) > + > +(cl:defconstant DB_RMW #x40000000) > + > +(cl:defconstant DB_BUFFER_SMALL -30999) > + > +(cl:defconstant DB_DONOTINDEX -30998) > + > +(cl:defconstant DB_KEYEMPTY -30997) > + > +(cl:defconstant DB_KEYEXIST -30996) > + > +(cl:defconstant DB_LOCK_DEADLOCK -30995) > + > +(cl:defconstant DB_LOCK_NOTGRANTED -30994) > + > +(cl:defconstant DB_LOG_BUFFER_FULL -30993) > + > +(cl:defconstant DB_NOSERVER -30992) > + > +(cl:defconstant DB_NOSERVER_HOME -30991) > + > +(cl:defconstant DB_NOSERVER_ID -30990) > + > +(cl:defconstant DB_NOTFOUND -30989) > + > +(cl:defconstant DB_OLD_VERSION -30988) > + > +(cl:defconstant DB_PAGE_NOTFOUND -30987) > + > +(cl:defconstant DB_REP_DUPMASTER -30986) > + > +(cl:defconstant DB_REP_HANDLE_DEAD -30985) > + > +(cl:defconstant DB_REP_HOLDELECTION -30984) > + > +(cl:defconstant DB_REP_IGNORE -30983) > + > +(cl:defconstant DB_REP_ISPERM -30982) > + > +(cl:defconstant DB_REP_JOIN_FAILURE -30981) > + > +(cl:defconstant DB_REP_LEASE_EXPIRED -30980) > + > +(cl:defconstant DB_REP_LOCKOUT -30979) > + > +(cl:defconstant DB_REP_NEWSITE -30978) > + > +(cl:defconstant DB_REP_NOTPERM -30977) > + > +(cl:defconstant DB_REP_UNAVAIL -30976) > + > +(cl:defconstant DB_RUNRECOVERY -30975) > + > +(cl:defconstant DB_SECONDARY_BAD -30974) > + > +(cl:defconstant DB_VERIFY_BAD -30973) > + > +(cl:defconstant DB_VERSION_MISMATCH -30972) > + > +(cl:defconstant DB_ALREADY_ABORTED -30899) > + > +(cl:defconstant DB_DELETED -30898) > + > +(cl:defconstant DB_EVENT_NOT_HANDLED -30897) > + > +(cl:defconstant DB_NEEDSPLIT -30896) > + > +(cl:defconstant DB_REP_BULKOVF -30895) > + > +(cl:defconstant DB_REP_EGENCHG -30894) > + > +(cl:defconstant DB_REP_LOGREADY -30893) > + > +(cl:defconstant DB_REP_NEWMASTER -30892) > + > +(cl:defconstant DB_REP_PAGEDONE -30891) > + > +(cl:defconstant DB_SURPRISE_KID -30890) > + > +(cl:defconstant DB_SWAPBYTES -30889) > + > +(cl:defconstant DB_TIMEOUT -30888) > + > +(cl:defconstant DB_TXN_CKP -30887) > + > +(cl:defconstant DB_VERIFY_FATAL -30886) > +(cl:defconstant DB_DBT_APPMALLOC #x001 "Callback allocated > memory.") > +(cl:defconstant DB_DBT_DUPOK #x002 "Insert if > duplicate.") > +(cl:defconstant DB_DBT_ISSET #x004 "Lower level calls > set value.") > +(cl:defconstant DB_DBT_MALLOC #x008 "Return in malloc'd > memory.") > +(cl:defconstant DB_DBT_MULTIPLE #x010 "References multiple > records.") > +(cl:defconstant DB_DBT_PARTIAL #x020 "Partial put/get.") > +(cl:defconstant DB_DBT_REALLOC #x040 "Return in > realloc'd memory.") > +(cl:defconstant DB_DBT_USERCOPY #x080 "Use the user- > supplied callback.") > +(cl:defconstant DB_DBT_USERMEM #x100 "Return in user's > memory.") > + > +(defconstant DB-BTREE 1) > +(defconstant DB-HASH 2) > +(defconstant DB-RECNO 3) > +(defconstant DB-QUEUE 4) > +(defconstant DB-UNKNOWN 5) > +(defconstant DB_SEQ_DEC #x00000001) > +(defconstant DB_SEQ_INC #x00000002) > +(defconstant DB_SEQ_RANGE_SET #x00000004) > +(defconstant DB_SEQ_WRAP #x00000008) > +(defconstant DB_SEQ_WRAPPED #x00000010) > hunk ./src/db-bdb/berkeley-db.lisp 551 > -(defconstant DB_LOCK_DEFAULT 1) > -(defconstant DB_LOCK_EXPIRE 2) > -(defconstant DB_LOCK_MAXLOCKS 3) > -(defconstant DB_LOCK_MAXWRITE 4) > -(defconstant DB_LOCK_MINLOCKS 5) > -(defconstant DB_LOCK_MINWRITE 6) > -(defconstant DB_LOCK_OLDEST 7) > -(defconstant DB_LOCK_RANDOM 8) > -(defconstant DB_LOCK_YOUNGEST 9) > } > > Context: > > [db-postmodern: optimized map-index for -by-value case > alex.mizrahi at gmail.com**20071207195402] > [db-postmodern: optimized form-slot-key for persistent-slot-reader > alex.mizrahi at gmail.com**20071207200835 > it uses SBCL internal function now, for other implementation it's > less optimized. > ] > [db-postmodern: small example update > alex.mizrahi at gmail.com**20071207200630] > [added sh script for flushing logs sample > alex.mizrahi at gmail.com**20070920095806] > [db-postmodern removed possiblity of using NIL as a key in btrees > Henrik Hjelte**20071124163828] > [cursor-duplicate removed from db-postmodern > Henrik Hjelte**20071124163701] > [removed a little compiler warning (typo) > Henrik Hjelte**20071122151929] > [remove kind-hints parameter from add-index > Henrik Hjelte**20071122151046 > Probably a coming feature from Ian, but > right now it breaks the generic function add-index > and thus postmodern, so I removed it for now. > ] > [Ensure set-db-synch is defined before pset is loaded > sross at common-lisp.net**20071214145041] > [Fix instance deserialization to bypass initialization protocol > sross at common-lisp.net**20071214141938] > [Fix to from-end traversal of new map-index > eslick at common-lisp.net**20071130223524] > [New map-index implementation > eslick at common-lisp.net**20071130222620] > [Cheaper get-instance-by-value > eslick at common-lisp.net**20071130222520] > [TAG ELEPHANT-0-9-1 > ieslick at common-lisp.net**20071116153634] > Patch bundle hash: > 1d34a40a2f63dd49153a97251e2959cec6d7137d > > > -- > With best regards, > Anton Kazennikov. mailto:kazennikov[at]mirea.ru ICQ# 98965967 > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From kazennikov.nntp at gmail.com Sun Dec 30 17:29:34 2007 From: kazennikov.nntp at gmail.com (Anton Kazennikov) Date: Sun, 30 Dec 2007 20:29:34 +0300 Subject: [elephant-devel] Re: patch: update db-bdb to Berkeley DB 4.6 References: <87ejd4dvzg.fsf@kzn.homelinux.org> Message-ID: <87lk7c3xc1.fsf@kzn.homelinux.org> Ian Eslick writes: > Great, thank you for the patches! The update from BDB 4.x to 4.(x+1) > have all had this constant problem. At some point we should figure out > how to generate the lisp code that matches this file so we don't have > to do this again and can link against any version... Hmm.. I think that I have an idea how to do this, it isn't quite mine, but from cl-berkeley-db: to use swig. Although, it has only cffi output, it might work well do generate constants definitions. > > But for now we'll release this patch in contrib so people using the > development branch can run 4.5 (default) or 4.6 via the patch. That's right - I haven't test it extensively, I think it would work well as is, but bugs are subtle things :-) -- With best regards, Anton Kazennikov. mailto:kazennikov[at]gmail.com ICQ# 98965967