From leslie.polzer at gmx.net Sat Aug 2 06:49:00 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 2 Aug 2008 08:49:00 +0200 (CEST) Subject: [elephant-devel] Re: Deferred indexing In-Reply-To: <61376.88.73.253.202.1216985559.squirrel@mail.stardawn.org> References: <61376.88.73.253.202.1216985559.squirrel@mail.stardawn.org> Message-ID: <63139.88.73.235.214.1217659740.squirrel@mail.stardawn.org> > It would be nice to be able to say: > > (defpclass banana () (:index t)) > > (setf banana (make-instance 'banana)) > > ; do stuff with banana, then later optionally: > > (store-indexed-object banana) > > We could then do without proxy objects in Weblocks. (defun make-deferred-instance (classname &rest initargs) (with-transaction () (let ((obj (apply #'make-instance classname initargs))) (remove-kv (slot-value obj 'elephant::%oid) (find-class-index classname))))) (defun activate-deferred-instance (obj) (let ((oid (slot-value obj 'elephant::%oid)) (idx (find-class-index (class-of obj)))) (if (get-value oid idx) (warn "Attempted to activate Object ~S which is already indexed!" obj) (setf (get-value oid idx) obj)))) This seems to work nicely, even with secondary indices. Haven't tested derived, though. Leslie From killerstorm at newmail.ru Sat Aug 2 11:54:53 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 2 Aug 2008 14:54:53 +0300 Subject: [elephant-devel] Re: Deferred indexing References: <61376.88.73.253.202.1216985559.squirrel@mail.stardawn.org> <63139.88.73.235.214.1217659740.squirrel@mail.stardawn.org> Message-ID: ??>> We could then do without proxy objects in Weblocks. LPP> This seems to work nicely, even with secondary indices. that looks like an ugly hack, though. what problem are you trying to solve? hide some instances from enumeration? why not just make slot called "visible" so you can just filter out instances that are not visible? From leslie.polzer at gmx.net Sat Aug 2 12:15:05 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 2 Aug 2008 14:15:05 +0200 (CEST) Subject: [elephant-devel] Re: Deferred indexing In-Reply-To: References: <61376.88.73.253.202.1216985559.squirrel@mail.stardawn.org> <63139.88.73.235.214.1217659740.squirrel@mail.stardawn.org> Message-ID: <64160.88.73.235.214.1217679305.squirrel@mail.stardawn.org> > what problem are you trying to solve? > hide some instances from enumeration? why not just make slot called > "visible" so you can just filter out instances that are not visible? That would be another solution, but it's got a few features I don't like compared to the other one. As long as the ugly (actually not that ugly IMHO) hack doesn't have any negative side effects I'm fine. But thanks; I always value your opinion, Alex. Leslie From leslie.polzer at gmx.net Sat Aug 9 10:10:24 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 9 Aug 2008 12:10:24 +0200 (CEST) Subject: [elephant-devel] NIL values in secondary indices Message-ID: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org> (defpclass artichoke () ((slot1 :accessor slot1 :initform nil :index t))) -> # (make-instance 'artichoke) -> # (get-instances-by-range 'artichoke nil nil) -> NIL Is there any sense behind this result? Are NIL values somehow special to the secondary indexing mechanism? How should I find the missing instances with GET-INSTANCES-BY-RANGE? Leslie From killerstorm at newmail.ru Sat Aug 9 11:18:08 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 9 Aug 2008 14:18:08 +0300 Subject: [elephant-devel] Re: NIL values in secondary indices References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org> Message-ID: LPP> Is there any sense behind this result? Are NIL values somehow LPP> special to the secondary indexing mechanism? it is somehow special in map-index, and inherently in GET-INSTANCES-BY-RANGE -- if start is NIL it starts from beginning, it end is NIL it iterates to the end, so (NIL, NIL) range should mean all the sequence. behaviour you're observing is wrong, i guess. LPP> How should I find the missing instances with LPP> GET-INSTANCES-BY-RANGE? question is _why_ should you use GET-INSTANCES-BY-RANGE when you can do this via GET-INSTANCES-BY-VALUE? From leslie.polzer at gmx.net Sat Aug 9 12:14:15 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 9 Aug 2008 14:14:15 +0200 (CEST) Subject: [elephant-devel] Re: NIL values in secondary indices In-Reply-To: References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org> Message-ID: <64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> Hi Alex, > LPP> Is there any sense behind this result? Are NIL values somehow > LPP> special to the secondary indexing mechanism? > > it is somehow special in map-index, and inherently in > GET-INSTANCES-BY-RANGE -- > if start is NIL it starts from beginning, it end is NIL it iterates to the > end, so (NIL, NIL) > range should mean all the sequence. That's the behaviour per the documentation as I understand it, yes. > LPP> How should I find the missing instances with > LPP> GET-INSTANCES-BY-RANGE? > > question is _why_ should you use GET-INSTANCES-BY-RANGE when > you can do this via GET-INSTANCES-BY-VALUE? Because I need sorting... ideally the NIL values should just be clustered at the end or beginning of the sorted space. Leslie From killerstorm at newmail.ru Sat Aug 9 14:59:58 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sat, 9 Aug 2008 17:59:58 +0300 Subject: [elephant-devel] Re: NIL values in secondary indices References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org> <64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> Message-ID: ??>> question is _why_ should you use GET-INSTANCES-BY-RANGE when ??>> you can do this via GET-INSTANCES-BY-VALUE? LPP> Because I need sorting... ideally the NIL values should just be LPP> clustered at the end or beginning of the sorted space. it seems you're using db-postmodern, i have bad news for you -- postmodern backend supports correct sorting _only_ if all values are either interger, or string. if you ever insert NIL there, sorting will be random. yep, i know it sucks, but it's a price we pay for using SQL directly. actually supporting slot type (OR STRING NULL) is easier than supporting arbitrary type combinations -- we can translate NIL to SQL NULL, so NIL values will be clustered, and rest will be sorted, but it's still considerable amount of work, and it's tricky to support all cases.. LPP>> Is there any sense behind this result? Are NIL values somehow special to the secondary indexing mechanism? i've found why it's such -- there are actually checks in btree code, introduced by patch: [db-postmodern removed possiblity of using NIL as a key in btrees Henrik Hjelte**20071124163828] it works in pretty nasty way -- if key is NIL it silently ignores insert into btree. a problem is if NIL wasn't ignored it will upgrade btree key type to "object" -- that means that sorting won't work. iirc Henrik was annoyed with this behaviour, so he've just banned all NILs. this means GET-INSTANCES-BY-VALUE won't work either -- those object simply won't be in indices. this appears to be pretty bad, i'll check if i can fix this.. From leslie.polzer at gmx.net Sat Aug 9 19:35:04 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 9 Aug 2008 21:35:04 +0200 (CEST) Subject: [elephant-devel] Re: NIL values in secondary indices In-Reply-To: References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org> <64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> Message-ID: <62576.88.73.221.58.1218310504.squirrel@mail.stardawn.org> > iirc Henrik was annoyed with this behaviour, > so he've just banned all NILs. this means GET-INSTANCES-BY-VALUE > won't work either -- those object simply won't be in indices. > > this appears to be pretty bad, i'll check if i > can fix this.. That would be a good start. I can sort out the objects with NIL values later on the Lisp side -- that's acceptable. But those objects not showing up at all in the index is pretty bad indeed! Leslie From leslie.polzer at gmx.net Sun Aug 10 08:13:17 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 10 Aug 2008 10:13:17 +0200 (CEST) Subject: [elephant-devel] Re: NIL values in secondary indices In-Reply-To: References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org> <64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> Message-ID: <61837.88.73.224.67.1218355997.squirrel@mail.stardawn.org> While we're at it, I remember running into a similar problem with float slots. Are there any hidden difficulties in implementing float sort support? From killerstorm at newmail.ru Mon Aug 11 13:26:17 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 11 Aug 2008 16:26:17 +0300 Subject: [elephant-devel] Re: NIL values in secondary indices References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org><64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> <61837.88.73.224.67.1218355997.squirrel@mail.stardawn.org> Message-ID: LPP> While we're at it, I remember running into a similar problem LPP> with float slots. yep, if it sees float it will upgrade column to "object" type and sorting will be lost LPP> Are there any hidden difficulties in implementing float sort LPP> support? i think there are no, we just didn't get to it as floats aren't used as frequently as integers and strings are. i think it would be enough to modify functions postgres-format and postgres-value-to-lisp to support floats and it will work. From killerstorm at newmail.ru Mon Aug 11 13:44:49 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Mon, 11 Aug 2008 16:44:49 +0300 Subject: [elephant-devel] Re: NIL values in secondary indices References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org><64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> <62576.88.73.221.58.1218310504.squirrel@mail.stardawn.org> Message-ID: LPP> I can sort out the objects with NIL values later on the Lisp LPP> side -- that's acceptable. But those objects not showing up LPP> at all in the index is pretty bad indeed! if you don't need sorting you can just call get-instances-by-class and sort stuff manually. i've taken one more look at cursor code, it seems representing NIL as NULL is possible, but it will make code quite hairy, as SQL treats NULLs differently from normal values, it needs sort of two modes of operation. so i'm not sure if it's worth complicating code with this -- it might be cleaner to implement more general solution instead, to keep different types in different tables. but it might be simplier to support nils on map-index level rather than on btree/cursor level -- so get-instances-by-value will do SQL query like this: SELECT * FROM class_index_x WHERE class_index_x.key NOT IN (SELECT key FROM secondary_index_x) and get-instances-by-range can detect case when range includes NIL values and append append aforementioned query results to range results. so there will be no nulls in secondary indices, but they will be visible to class indexing APIs. drawback of this "workaround" solution is that it can be slow when table goes large, as it needs to traverse whole index. so there are three options how to do this: * via NULLs and hairy cursor code * major restructuring to support multiple types * workaround to support nils on indexing API level, with some overhead From leslie.polzer at gmx.net Mon Aug 11 13:54:43 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 11 Aug 2008 15:54:43 +0200 (CEST) Subject: [elephant-devel] Re: NIL values in secondary indices In-Reply-To: References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org><64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> <61837.88.73.224.67.1218355997.squirrel@mail.stardawn.org> Message-ID: <63213.88.73.198.233.1218462883.squirrel@mail.stardawn.org> > i think there are no, we just didn't get to it as floats aren't used > as frequently as integers and strings are. i think it would be > enough to modify functions postgres-format and postgres-value-to-lisp > to support floats and it will work. I don't have an immediate need for floats as sort keys, but I'm not really happy with this thing lurking in the shadows to jump at me at some inconvenient moment, so I'll try to implement this. Thanks a lot for the pointers. Leslie From leslie.polzer at gmx.net Mon Aug 11 14:01:21 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 11 Aug 2008 16:01:21 +0200 (CEST) Subject: [elephant-devel] Re: NIL values in secondary indices In-Reply-To: References: <62581.88.73.221.58.1218276624.squirrel@mail.stardawn.org><64795.88.73.221.58.1218284055.squirrel@mail.stardawn.org> <62576.88.73.221.58.1218310504.squirrel@mail.stardawn.org> Message-ID: <64941.88.73.198.233.1218463281.squirrel@mail.stardawn.org> > if you don't need sorting you can just call get-instances-by-class and > sort stuff manually. Not really acceptable for me, unfortunately -- or at least I guess so. What would the speed penalty be if I'd sort everything on the Lisp side? > i've taken one more look at cursor code, it seems representing > NIL as NULL is possible, but it will make code quite hairy, as SQL > treats NULLs differently from normal values, it needs sort of two modes > of operation. Some added IFNULL clauses won't help, I guess? > so i'm not sure if it's worth complicating code with this > -- it might be cleaner to implement more general solution instead, to > keep different types in different tables. Sounds like a major refactoring, or at least one for which I don't have the resources right now. > but it might be simplier to support nils on map-index level rather than > on btree/cursor level -- so get-instances-by-value will do SQL > query like this: > > SELECT * FROM class_index_x WHERE class_index_x.key > NOT IN (SELECT key FROM secondary_index_x) > > and get-instances-by-range can detect case when range includes NIL > values and append append aforementioned query results to range results. > so there will be no nulls in secondary indices, but they will be visible to > class indexing APIs. Well, at least it would be cheaper than doing the set difference on the Lisp side. What solution(s) could you afford to do right now? I could ponder working on the refactoring solution if we can somehow divide the work and when I know better what it would involve. Leslie From klists at saphor.de Wed Aug 13 06:06:12 2008 From: klists at saphor.de (Marc) Date: Wed, 13 Aug 2008 08:06:12 +0200 Subject: [elephant-devel] changing association slots In-Reply-To: <085bb9999533a805dde6410a51c7dc84.squirrel@webmailer.fh-worms.de> References: <00e045912595cb82e5a88cbfbe39c2d3.squirrel@webmailer.fh-worms.de> <6DE32DC0-264E-4ED9-AEA8-635816A5C637@media.mit.edu> <0d6933b7b0e516658500c12db144490b.squirrel@webmailer.fh-worms.de> <3f6f4c64c487d823074f6411d229ad35.squirrel@webmailer.fh-worms.de> <085bb9999533a805dde6410a51c7dc84.squirrel@webmailer.fh-worms.de> Message-ID: <48A279D4.2050907@saphor.de> Hi! > i forgot to attach the patch to the mail - so other users - if they have > the same problem - will be able to compare the current source with our > patch. > > here a slighly expanded version of the same patch that also allows remove-association to work for unbound slots. Best regards, Marc > > > >> hi, >> >> while we were working on the problem "associations between instances of >> the same class" we also were trying to solve the remove-association >> problem. >> >> we have a suggestion - it differs a little from the original way, but >> maybe it is helpful. >> >> so i sent a patch today. >> >> >> regards >> >> lukas >> >> >> >>> I was getting too much spam so require authentication on the Trac >>> now. The admins can be slow to grant accounts or enable Trac. If you >>> send me a report directly on e-mail I'll add it for you. >>> >>> Ian >>> >>> On Jul 9, 2008, at 3:55 PM, LukasGiessmann wrote: >>> >>> >>>> Hi Ian, >>>> >>>> I know this is not the right space to place this question butI don?t >>>> know >>>> were else I should do it. >>>> Tried to get an account for http://trac.common-lisp.net/elephant; I >>>> wrote >>>> a mail to admin at common-lisp.net but haven?t got an answer yet. >>>> Is there another possibility to mark this bug? >>>> >>>> Best regards >>>> Lukas >>>> >>>> >>>>> I haven't looked at that code in a bit, but I suspect that the right >>>>> thing happens if you just remove the association definition from >>>>> defclass and reevaluate the defclass form. Can you mark this as a >>>>> bug >>>>> in elephant so I can remember to look into it later? >>>>> >>>>> Thank you, >>>>> Ian >>>>> >>>>> On Jul 4, 2008, at 4:24 AM, LukasGiessmann wrote: >>>>> >>>>> >>>>>> hello, >>>>>> >>>>>> since our last mail working with associations turned out to go quite >>>>>> well >>>>>> but one more problem came up: >>>>>> i tried to change the associations by removing it and replacing it >>>>>> by a >>>>>> new association. but the function "remove-association" only works >>>>>> partially. >>>>>> >>>>>> 1. in this one-to-one association the function doesn't work at all. >>>>>> --- >>>>>> (defpclass job() >>>>>> ((title :initarg :title :accessor title-of :index t) >>>>>> (company :initarg :company :accessor company-of :index t) >>>>>> (job-holder :initarg :job-holder :accessor job-holder-of :associate >>>>>> person))) >>>>>> >>>>>> (defpclass person() >>>>>> ((name :initarg :name :accessor name-of :index t) >>>>>> (job :initarg :job :accessor job-of :associate job))) >>>>>> --- >>>>>> >>>>>> an error message shows up: >>>>>> >>>>>> (remove-association *job1* 'job-holder *person1*) ; -> There is no >>>>>> applicable method for the generic function #>>>>> FUNCTION >>>>>> ELEPHANT::OID (1)> when called with arguments (NIL). >>>>>> >>>>>> same thing happens when i use the following statement: >>>>>> (setf (job-holder-of *job1*) nil) >>>>>> >>>>>> >>>>>> 2. by one-to-many associations generally nothing happens at all. >>>>>> --- >>>>>> (defpclass job() >>>>>> ((title :initarg :title :accessor title-of :index t) >>>>>> (company :initarg :company :accessor company-of :index t) >>>>>> (job-holders :accessor job-holders-of :associate (person job)))) >>>>>> >>>>>> (defpclass person() >>>>>> ((name :initarg :name :accessor name-of :index t) >>>>>> (job :initarg :job :accessor job-of :associate job))) >>>>>> --- >>>>>> >>>>>> it only works in this case: >>>>>> --- >>>>>> (drop-instance *person2*) >>>>>> (remove-association *job2* 'job-holders *person2*) >>>>>> --- >>>>>> >>>>>> >>>>>> 3. in the case of many-to-many associations everything goes fine. >>>>>> >>>>>> how do i get the "remove-association" function to work? >>>>>> are there other ways for changing associations? >>>>>> >>>>>> >>>>>> thank you for your help >>>>>> >>>>>> lukas >>>>>> >>>>>> _______________________________________________ >>>>>> elephant-devel site list >>>>>> elephant-devel at common-lisp.net >>>>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>>>>> >>>>> _______________________________________________ >>>>> elephant-devel site list >>>>> elephant-devel at common-lisp.net >>>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>>>> >>>>> >>>> _______________________________________________ >>>> elephant-devel site list >>>> elephant-devel at common-lisp.net >>>> http://common-lisp.net/mailman/listinfo/elephant-devel >>>> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >>> >>> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- A non-text attachment was scrubbed... Name: remove-association.patch.gz Type: application/x-compressed-tar Size: 6127 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 20080813055629-e1a60-e712c0fd8ac5c33c82ec00cfdcb29314d774d158.gz Type: application/x-compressed-tar Size: 471 bytes Desc: not available URL: From leslie.polzer at gmx.net Wed Aug 27 18:08:16 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Wed, 27 Aug 2008 20:08:16 +0200 (CEST) Subject: [elephant-devel] Problems reloading Elephant system In-Reply-To: <7F01A3FF-2BC7-480D-B7F2-850E96D9D540@media.mit.edu> References: <61322.88.73.200.98.1216910564.squirrel@mail.stardawn.org> <7F01A3FF-2BC7-480D-B7F2-850E96D9D540@media.mit.edu> Message-ID: <61593.88.73.223.167.1219860496.squirrel@mail.stardawn.org> > I've noticed some of that stuff going on in elephant-unstable and > haven't been able to track it down yet. I'm surprised to hear it's in > stable... Any hints as to what kind of data this is failing on? To find the cause of this it would probably be beneficial to perform the split from the %DESERIALIZE blob into a generic function... Leslie From leslie.polzer at gmx.net Thu Aug 28 14:04:04 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Thu, 28 Aug 2008 16:04:04 +0200 (CEST) Subject: [elephant-devel] DB_LOCK_DEADLOCK support Message-ID: <64898.88.73.196.99.1219932244.squirrel@mail.stardawn.org> Is it correct that db4.7 has built-in deadlock detection, and elephant-unstable has been adapted to that? Leslie From eslick at media.mit.edu Thu Aug 28 14:22:40 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 28 Aug 2008 10:22:40 -0400 Subject: [elephant-devel] DB_LOCK_DEADLOCK support In-Reply-To: <64898.88.73.196.99.1219932244.squirrel@mail.stardawn.org> References: <64898.88.73.196.99.1219932244.squirrel@mail.stardawn.org> Message-ID: <3D3F566D-BEF1-4A4A-A04C-C627B775DED9@media.mit.edu> You can run a thread which aborts threads that otherwise would have hung. "The DB_ENV->failchk method can now abort transactions for threads, which have failed while blocked on a concurrency lock. This significantly decreases the need for database environment recovery after thread of control failure. [#15626]" My understanding is that you have to launch a thread to perform these checks and cause the transactions to abort (this requires true multi- threading - Allegro for example would not benefit from this since it still uses a semi-cooperative multitasking to avoid all locking that has been put into SBCL). I've made some code changes in elephant-unstable to support 4.7 but had some bugs getting it running (problems with constants, API changes) but haven't had the chance to track the errors down. I won't be doing any more elephant development until at least mid-September. I am aware that there is a reasonable set of issues with various lisps, an intermittent de-serializer bug that also shows up in stable, and a condition that requires restarting the db when I kill stuck threads that are in the middle of a transaction... Ian On Aug 28, 2008, at 10:04 AM, Leslie P. Polzer wrote: > > Is it correct that db4.7 has built-in deadlock detection, > and elephant-unstable has been adapted to that? > > Leslie > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel