From killerstorm at newmail.ru Sun May 1 18:36:28 2011 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 1 May 2011 21:36:28 +0300 Subject: [elephant-devel] inherited indices References: Message-ID: IE> One solution is to get rid of the :inherited option and simply allow a IE> :subclasses? option to get-instances-by-value/range that uses the mop IE> to find subclass indexes and does a merge sort of all objects returned IE> by the various indexes. This becomes time O(k log_b n) and gets rid of IE> inherited indexing. Yes, but it doesn't help with iteration with map-... functions and cursor, e.g. when you want latest 10 items or something like that. Reading all items just to get 10 latest will have a huge performance impact. It is possible to traverse multiple cursors at once, but it is weird and slow. So I think while :inherit is not perfect it is a good tool to achieve desired results, especially together with derived indices -- you can do pretty much whatever you want, just API is less than perfect. IE> However, if the k is going to matter or you want a faster option, you IE> can keep :inherited (one big index) and have IE> get-instances-by--value/range filter out classes you didn't ask for IE> (i.e if base class get everything, if asking for subclass and inherited IE> then just return matching subclasses - you could even do this before IE> object instantiation to make it extremely efficient by testing the IE> schema ID associated with the OID (checking for older schemas that need IE> to be upgraded if schema upgrading is set to lazy) I think let's start on a simple approach (filtering using typep) and then maybe implement a faster one. From killerstorm at newmail.ru Sun May 1 19:08:27 2011 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 1 May 2011 22:08:27 +0300 Subject: [elephant-devel] How do I populate an index of a slot that wascreated later with values created before slot became indexed? References: <852EF08C-2473-42A2-BA82-A2715F1069A7@media.mit.edu> Message-ID: IE> Leslie and Alex are both capable of making changes and improvements and IE> I'm happy to give checkin privs if they're interested (I think Leslie IE> has them already). Yep, I'd like to get checkin privs too :) IE> Also, if someone wants to step up as maintainer I'm happy to step IE> down, but will hold the fort as best I can otherwise. I would consider a position of co-maintainer -- I'm not sure about a long run, but I could work more on Elephant for a year or so. To be clear I don't have a strategic vision at moment, but I'd like to focus on stability and usability improvements. The thing is we're going to switch to version 1.0 in stix.to (finally!) and so both I and Henrik are interested in getting Elephant stable, fast and usable. I'm also going to try BDB Elephant backed in my own personal small projects so it won't be left forgotten. Now regardless of whether I'm becoming a co-maintainer or not, what is our policy on changing things? Bugfixing and small improvements won't be a problem I guess, but if some API or major thing needs to be changed? Obviously it is a good idea to discuss it before implementing, but I'm not sure what counts as consensus etc. For example, I don't like that (setf slot-value) on association slots does add-association while slot-value returns a list of instances. This means that (setf (slot-value instance 'assoc) (slot-value instance 'assoc)) will signal an error because types are not compatible. I think this violates behaviour one expects from CLOS class. There is a similar API for set-valued slots, but in that case it works like an extension and doesn't break compatibility. From eslick at media.mit.edu Mon May 2 17:19:27 2011 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 2 May 2011 10:19:27 -0700 Subject: [elephant-devel] How do I populate an index of a slot that wascreated later with values created before slot became indexed? In-Reply-To: References: <852EF08C-2473-42A2-BA82-A2715F1069A7@media.mit.edu> Message-ID: <752E6103-1677-47C8-B775-29530EF80D24@media.mit.edu> FYI - I still run three production sites on Elephant BDB, although I haven't been tracking Oracle's latest releases... I'll upgrade at least one of them to the next version you release and do some testing. Elephant has been in pre-release status for a 1.0 release for almost three years now, it would be fantastic to clean up the remaining API, testing, and stability issues. That would be a great co-maintainer contribution. I was also unhappy with the association slot API so don't use them myself and there are other awkward issues here and there that kept me from declaring a 1.0 release. As far as process is concerned; just run architecture or API change proposals by the list before you make them. I used to consider silence as assent. I will chime in as I can on likely consequences of changes or render an opinion, but feel free to use your own judgement. I would recommend a complete code review of the schema implementation as a good first step; that's one area that often causes problems when making changes to different slot types. I'm sure you'll catch design or implementation issues that I missed at the time. Thanks, Ian On May 1, 2011, at 12:08 PM, Alex Mizrahi wrote: > IE> Leslie and Alex are both capable of making changes and improvements and > IE> I'm happy to give checkin privs if they're interested (I think Leslie > IE> has them already). > > Yep, I'd like to get checkin privs too :) > > IE> Also, if someone wants to step up as maintainer I'm happy to step > IE> down, but will hold the fort as best I can otherwise. > > I would consider a position of co-maintainer -- I'm not sure about a long > run, but I could work more on Elephant for a year or so. > To be clear I don't have a strategic vision at moment, but I'd like to focus > on stability and usability improvements. > > The thing is we're going to switch to version 1.0 in stix.to (finally!) and > so both I and Henrik are interested in getting Elephant stable, fast and > usable. > I'm also going to try BDB Elephant backed in my own personal small projects > so it won't be left forgotten. > > Now regardless of whether I'm becoming a co-maintainer or not, what is our > policy on changing things? > Bugfixing and small improvements won't be a problem I guess, but if some API > or major thing needs to be changed? > Obviously it is a good idea to discuss it before implementing, but I'm not > sure what counts as consensus etc. > > For example, I don't like that (setf slot-value) on association slots does > add-association while slot-value returns a list of instances. > This means that > > (setf (slot-value instance 'assoc) (slot-value instance 'assoc)) > > will signal an error because types are not compatible. I think this violates > behaviour one expects from CLOS class. > > There is a similar API for set-valued slots, but in that case it works like > an extension and doesn't break compatibility. > > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lukas.georgieff at hotmail.com Wed May 4 09:34:46 2011 From: lukas.georgieff at hotmail.com (Lukas Georgieff) Date: Wed, 4 May 2011 11:34:46 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances Message-ID: Hi, I want to use one elephant (BDB) store that is used with two instances of sbcl in parallel. The documentation [1] says that "Transactions will ensure there is no interaction between processes.". Does this mean that I have to use the common methods to open the store (open-store '(:BDB "path-to-database")) and have to ensure that all database-operations are guarded by "with-transaction"? Some time ago there was a mail with the subject "[elephant-devel] PANIC: fatal region error detected; run recovery?" - I think it was a similar problem, Leslie answered that > when using multiple processes with one Elephant BDB store> you need to open the store with :register t. that means "(open-store '(:BDB "path-to-database") :register t)" - I did not find this in the documentation? Has someone any experience using elephant in that manner? Best regards, Lukas 1: http://common-lisp.net/project/elephant/doc/elephant.html#Multiple-Processes-and-Distributed-Applications -------------- next part -------------- An HTML attachment was scrubbed... URL: From sky at viridian-project.de Wed May 4 16:23:20 2011 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 4 May 2011 18:23:20 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: Message-ID: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> Hi Lukas, > Has someone any experience using elephant in that manner? Yes -- lots in fact. But you need to ask a specific question. Leslie From lukas.georgieff at hotmail.com Wed May 4 18:37:51 2011 From: lukas.georgieff at hotmail.com (Lukas Georgieff) Date: Wed, 4 May 2011 20:37:51 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> Message-ID: Hi Leslie, >Hi Lukas, > >> Has someone any experience using elephant in that manner? > >Yes -- lots in fact. But you need to ask a specific question. well, if the points I've mentioned before are correct, I've only one specific question :-) We want to share a network device that contains the Berkeley-DB files. This BDB shall be accessed by two or more other machines that execute a SBCL interpreter with our program. I am not sure if this works similar to executing two SBCL instances on one machine, because I don't know if the elephant-bdb-locking works Across several machines! Thanks in advance, Lukas From sky at viridian-project.de Wed May 4 18:44:05 2011 From: sky at viridian-project.de (Leslie P. Polzer) Date: Wed, 4 May 2011 20:44:05 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> Message-ID: <20110504184405.GA21390@viridian-project.de> On Wed, May 04, 2011 at 08:37:51PM +0200, Lukas Georgieff wrote: > well, if the points I've mentioned before are correct, I've only one > specific question :-) > > We want to share a network device that contains the Berkeley-DB files. > This BDB shall be accessed by two or more other machines that execute a SBCL > interpreter with our program. > > I am not sure if this works similar to executing two SBCL instances on one > machine, because I don't know if the elephant-bdb-locking works > Across several machines! Are you talking about NFS? I've never done anything like that. That's a pure BDB issue though. Leslie From eslick at media.mit.edu Wed May 4 18:51:20 2011 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 4 May 2011 11:51:20 -0700 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> Message-ID: BDB does not work across multiple machines, its transaction support relies on a shared memory region to handle locking so any shared memory architecture (with multiple CPUs) works fine but separate machines and a NAS device don't work. You'll have to use the Postgres store (5x slower, but reasonably scalable) if you want concurrent access to a single Elephant DB from multiple machines. On May 4, 2011, at 11:37 AM, Lukas Georgieff wrote: > Hi Leslie, > >> Hi Lukas, >> >>> Has someone any experience using elephant in that manner? >> >> Yes -- lots in fact. But you need to ask a specific question. > > well, if the points I've mentioned before are correct, I've only one > specific question :-) > > We want to share a network device that contains the Berkeley-DB files. > This BDB shall be accessed by two or more other machines that execute a SBCL > interpreter with our program. > > I am not sure if this works similar to executing two SBCL instances on one > machine, because I don't know if the elephant-bdb-locking works > Across several machines! > > > Thanks in advance, > > Lukas > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Wed May 4 19:13:06 2011 From: eslick at media.mit.edu (Ian Eslick) Date: Wed, 4 May 2011 12:13:06 -0700 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> Message-ID: <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> I should clarify the first sentence. With a bunch of external machinery to support a distributed transaction model, it's possible to run BDB across machines, but Elephant doesn't have any of it so can only use the built-in shared memory locking facility. There was some talk a year or two ago about doing a distributed locking solution for Elephant, but I don't know the status of it. Thanks, Ian On May 4, 2011, at 11:51 AM, Ian Eslick wrote: > BDB does not work across multiple machines, its transaction support relies on a shared memory region to handle locking so any shared memory architecture (with multiple CPUs) works fine but separate machines and a NAS device don't work. You'll have to use the Postgres store (5x slower, but reasonably scalable) if you want concurrent access to a single Elephant DB from multiple machines. > > On May 4, 2011, at 11:37 AM, Lukas Georgieff wrote: > >> Hi Leslie, >> >>> Hi Lukas, >>> >>>> Has someone any experience using elephant in that manner? >>> >>> Yes -- lots in fact. But you need to ask a specific question. >> >> well, if the points I've mentioned before are correct, I've only one >> specific question :-) >> >> We want to share a network device that contains the Berkeley-DB files. >> This BDB shall be accessed by two or more other machines that execute a SBCL >> interpreter with our program. >> >> I am not sure if this works similar to executing two SBCL instances on one >> machine, because I don't know if the elephant-bdb-locking works >> Across several machines! >> >> >> Thanks in advance, >> >> Lukas >> >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lukas.georgieff at hotmail.com Wed May 4 20:21:02 2011 From: lukas.georgieff at hotmail.com (lukas.georgieff@hormail.com) Date: Wed, 4 May 2011 22:21:02 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> Message-ID: Hi Ian, hi Leslie, thank you for comments! > I should clarify the first sentence. With a bunch of external machinery to support a distributed transaction model, it's possible to run BDB across machines, but Elephant > doesn't have any of it so can only use the built-in shared memory locking facility. There was some talk a year or two ago about doing a distributed locking solution for > Elephant, but I don't know the status of it. @Leslie: > Are you talking about NFS? I've never done anything like that. That's a pure BDB issue though. Yes. Has anybody of you any idea or clue that could help us to use elephant with Berkeley DB in the mentioned scenario, maybe an additional framework, etc. that could handle the locking of the shared db? Sincerely, Lukas From henrik at evahjelte.com Thu May 5 10:01:40 2011 From: henrik at evahjelte.com (Henrik Hjelte) Date: Thu, 5 May 2011 12:01:40 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> Message-ID: > Has anybody of you any idea or clue that could help us to use elephant with > Berkeley DB in the mentioned scenario, maybe an additional framework, etc. > that could handle the locking of the shared db? > I am curious why the postgres backend is so unpopular. As far as I know there is no real data to support that it is slow in "real world" applications. Just running the testcases and measuring total time is not a very good performance test. It also has caching mechanisms that work in multi-process applications. For an application running on several computer is think network performance might be a bottleneck, why should berkeley db have an edge in that case? /Henrik Hjelte From lukas.georgieff at hotmail.com Thu May 5 10:10:08 2011 From: lukas.georgieff at hotmail.com (Lukas Georgieff) Date: Thu, 5 May 2011 12:10:08 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: , <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org>, , , <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu>, , Message-ID: Hi, we have been developing our application always on top of BDB, so it would be nice to get running it in the mentioned scenario with BDB.But seems that we've to rethink and test our app with postgres! Best regards, Lukas > Date: Thu, 5 May 2011 12:01:40 +0200 > From: henrik at evahjelte.com > To: elephant-devel at common-lisp.net > Subject: Re: [elephant-devel] using one store with mutiple lisp instances > > > Has anybody of you any idea or clue that could help us to use elephant with > > Berkeley DB in the mentioned scenario, maybe an additional framework, etc. > > that could handle the locking of the shared db? > > > > I am curious why the postgres backend is so unpopular. As far as I > know there is no real data to support that it is slow in "real world" > applications. Just running the testcases and measuring total time is > not a very good performance test. It also has caching mechanisms that > work in multi-process applications. For an application running on > several computer is think network performance might be a bottleneck, > why should berkeley db have an edge in that case? > > /Henrik Hjelte > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at media.mit.edu Thu May 5 15:11:38 2011 From: eslick at media.mit.edu (Ian Eslick) Date: Thu, 5 May 2011 08:11:38 -0700 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: , <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org>, , , <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu>, , Message-ID: <9A7E4167-6074-486A-AF6A-8CFB33AEB8EC@media.mit.edu> I definitely recommend the Postgres backend for your application. It will be a significant amount of tricky work to get BDB replication + distributed transactions to work cleanly with Elephant. Not worth the time / bug risks I think. It should be trivial to migrate. Henrik, with the caching you (or was it Alex?) put in, real world apps may perform just fine. I suspect BDB would have a performance edge as object slot reads that miss in the cache would be satisfied with a local disk seek vs. a network transaction and remote disk seek. I wouldn't consider distributed BDB until you hit a wall with Postgres and felt that 2x performance was a material benefit. Ian On May 5, 2011, at 3:10 AM, Lukas Georgieff wrote: > Hi, > > we have been developing our application always on top of BDB, so it would be nice to get running it in the mentioned scenario with BDB. > But seems that we've to rethink and test our app with postgres! > > > Best regards, > > Lukas > > > > > Date: Thu, 5 May 2011 12:01:40 +0200 > > From: henrik at evahjelte.com > > To: elephant-devel at common-lisp.net > > Subject: Re: [elephant-devel] using one store with mutiple lisp instances > > > > > Has anybody of you any idea or clue that could help us to use elephant with > > > Berkeley DB in the mentioned scenario, maybe an additional framework, etc. > > > that could handle the locking of the shared db? > > > > > > > I am curious why the postgres backend is so unpopular. As far as I > > know there is no real data to support that it is slow in "real world" > > applications. Just running the testcases and measuring total time is > > not a very good performance test. It also has caching mechanisms that > > work in multi-process applications. For an application running on > > several computer is think network performance might be a bottleneck, > > why should berkeley db have an edge in that case? > > > > /Henrik Hjelte > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.georgieff at hotmail.com Thu May 5 18:50:42 2011 From: lukas.georgieff at hotmail.com (Lukas Georgieff) Date: Thu, 5 May 2011 20:50:42 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: <9A7E4167-6074-486A-AF6A-8CFB33AEB8EC@media.mit.edu> References: , , <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org>, , , , , , <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu>, , , , , , <9A7E4167-6074-486A-AF6A-8CFB33AEB8EC@media.mit.edu> Message-ID: Hi, thank you all for your comments and ideas! > I definitely recommend the Postgres backend for your application This seems to be the solution :-) Best regards, Lukas From: eslick at media.mit.edu Date: Thu, 5 May 2011 08:11:38 -0700 To: elephant-devel at common-lisp.net Subject: Re: [elephant-devel] using one store with mutiple lisp instances I definitely recommend the Postgres backend for your application. It will be a significant amount of tricky work to get BDB replication + distributed transactions to work cleanly with Elephant. Not worth the time / bug risks I think. It should be trivial to migrate. Henrik, with the caching you (or was it Alex?) put in, real world apps may perform just fine. I suspect BDB would have a performance edge as object slot reads that miss in the cache would be satisfied with a local disk seek vs. a network transaction and remote disk seek. I wouldn't consider distributed BDB until you hit a wall with Postgres and felt that 2x performance was a material benefit. Ian On May 5, 2011, at 3:10 AM, Lukas Georgieff wrote:Hi, we have been developing our application always on top of BDB, so it would be nice to get running it in the mentioned scenario with BDB.But seems that we've to rethink and test our app with postgres! Best regards, Lukas > Date: Thu, 5 May 2011 12:01:40 +0200 > From: henrik at evahjelte.com > To: elephant-devel at common-lisp.net > Subject: Re: [elephant-devel] using one store with mutiple lisp instances > > > Has anybody of you any idea or clue that could help us to use elephant with > > Berkeley DB in the mentioned scenario, maybe an additional framework, etc. > > that could handle the locking of the shared db? > > > > I am curious why the postgres backend is so unpopular. As far as I > know there is no real data to support that it is slow in "real world" > applications. Just running the testcases and measuring total time is > not a very good performance test. It also has caching mechanisms that > work in multi-process applications. For an application running on > several computer is think network performance might be a bottleneck, > why should berkeley db have an edge in that case? > > /Henrik Hjelte > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From alex.mizrahi at gmail.com Sat May 7 16:20:09 2011 From: alex.mizrahi at gmail.com (Alex Mizrahi) Date: Sat, 07 May 2011 19:20:09 +0300 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> Message-ID: > I am curious why the postgres backend is so unpopular. I would not recommend postgres backend as a default choice because it has its own "peculiarities". Particularly, it requires a discipline with use of types, as only strings and (64-bit) integers are properly sorted. Another major drawback is that there is no GC for blobs so it requires export/import to reclaim space from time to time. But if using multiple machines is a requirement then it looks like postgres is the only option, as Oracle confirms that use of BDB DB on remote filesystem doesn't work for multiple clients: http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/env_remote.html > As far as I > know there is no real data to support that it is slow in "real world" > applications. There is certain overhead when working with blobs, so it is slower than it might be in theory. But whether it is "slow" is another question. DB can bottleneck in multiple places (disk operations, concurrency locks, network...) so there is no simple answer. To get a good performance with postgres backend one needs to understand that each DB-related operation means a DB query (which potentially goes over the network). So it is a good idea to structure code in a way that minimizes number of slot reads/writes, range queries etc. > It also has caching mechanisms that > work in multi-process applications. Yeah, this gives some freedom with slot reads (although still has some overhead comparing to CLOS slot reads). If we implement cache for range queries as well we can get to a situation where almost nothing is read from DB as long as there is enough memory for cache. This would have a huge impact of DB performance, I think. Another possible improvement is to reimplement cache synchronization with notifications instead of DB tables, as this might drastically reduce overhead of cache synchronization. (Update notifications won't be written to disk.) > why should berkeley db have an edge in that case? With improved caching probably postgres will have an edge for common applications. From eslick at media.mit.edu Sat May 7 17:34:42 2011 From: eslick at media.mit.edu (Ian Eslick) Date: Sat, 7 May 2011 10:34:42 -0700 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> Message-ID: <01CF8F20-3D1E-4FDE-B4CA-FDE957EC4A56@media.mit.edu> How can you guarantee that the postgres cache is coherent across machines? Also, BDB does caching in shared memory regions across multiple processes, so I'm not sure how much extra benefit you'll get from the Postgres cache. Sent from my iPhone On May 7, 2011, at 9:20 AM, Alex Mizrahi wrote: >> I am curious why the postgres backend is so unpopular. > > I would not recommend postgres backend as a default choice because it > has its own "peculiarities". Particularly, it requires a discipline with > use of types, as only strings and (64-bit) integers are properly sorted. > Another major drawback is that there is no GC for blobs so it requires > export/import to reclaim space from time to time. > > But if using multiple machines is a requirement then it looks like > postgres is the only option, as Oracle confirms that use of BDB DB on > remote filesystem doesn't work for multiple clients: > > http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/env_remote.html > >> As far as I >> know there is no real data to support that it is slow in "real world" >> applications. > > There is certain overhead when working with blobs, so it is slower than > it might be in theory. But whether it is "slow" is another question. DB > can bottleneck in multiple places (disk operations, concurrency locks, > network...) so there is no simple answer. > > To get a good performance with postgres backend one needs to understand > that each DB-related operation means a DB query (which potentially goes > over the network). So it is a good idea to structure code in a way that > minimizes number of slot reads/writes, range queries etc. > >> It also has caching mechanisms that >> work in multi-process applications. > > Yeah, this gives some freedom with slot reads (although still has some > overhead comparing to CLOS slot reads). > > If we implement cache for range queries as well we can get to a > situation where almost nothing is read from DB as long as there is > enough memory for cache. This would have a huge impact of DB > performance, I think. > > Another possible improvement is to reimplement cache synchronization > with notifications instead of DB tables, as this might drastically > reduce overhead of cache synchronization. (Update notifications won't be > written to disk.) > >> why should berkeley db have an edge in that case? > > With improved caching probably postgres will have an edge for common > applications. > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From lukas.georgieff at hotmail.com Sat May 7 20:24:27 2011 From: lukas.georgieff at hotmail.com (lukas.georgieff@hormail.com) Date: Sat, 7 May 2011 22:24:27 +0200 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> Message-ID: Hi, since the last mails I am are working on an idea of a federated DB - so we would use two separate DBs on two machines. The application is designed to synchronize the stored data over Atom Feeds (not because of the last problem, but as a main feature of the app). Since it is not critical to be always 100% synchronized this concept is an option. We will gain some benefits in parallel reading-time and availability. Additionally, we can use the our Backend without changes. Sincerely, Lukas -----Original Message----- From: Alex Mizrahi [mailto:alex.mizrahi at gmail.com] Sent: Samstag, 7. Mai 2011 18:20 To: elephant-devel at common-lisp.net Subject: Re: [elephant-devel] using one store with mutiple lisp instances > I am curious why the postgres backend is so unpopular. I would not recommend postgres backend as a default choice because it has its own "peculiarities". Particularly, it requires a discipline with use of types, as only strings and (64-bit) integers are properly sorted. Another major drawback is that there is no GC for blobs so it requires export/import to reclaim space from time to time. But if using multiple machines is a requirement then it looks like postgres is the only option, as Oracle confirms that use of BDB DB on remote filesystem doesn't work for multiple clients: http://download.oracle.com/docs/cd/E17076_02/html/programmer_reference/env_r emote.html > As far as I > know there is no real data to support that it is slow in "real world" > applications. There is certain overhead when working with blobs, so it is slower than it might be in theory. But whether it is "slow" is another question. DB can bottleneck in multiple places (disk operations, concurrency locks, network...) so there is no simple answer. To get a good performance with postgres backend one needs to understand that each DB-related operation means a DB query (which potentially goes over the network). So it is a good idea to structure code in a way that minimizes number of slot reads/writes, range queries etc. > It also has caching mechanisms that > work in multi-process applications. Yeah, this gives some freedom with slot reads (although still has some overhead comparing to CLOS slot reads). If we implement cache for range queries as well we can get to a situation where almost nothing is read from DB as long as there is enough memory for cache. This would have a huge impact of DB performance, I think. Another possible improvement is to reimplement cache synchronization with notifications instead of DB tables, as this might drastically reduce overhead of cache synchronization. (Update notifications won't be written to disk.) > why should berkeley db have an edge in that case? With improved caching probably postgres will have an edge for common applications. _______________________________________________ elephant-devel site list elephant-devel at common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel From alex.mizrahi at gmail.com Sun May 8 02:53:18 2011 From: alex.mizrahi at gmail.com (Alex Mizrahi) Date: Sun, 08 May 2011 05:53:18 +0300 Subject: [elephant-devel] using one store with mutiple lisp instances In-Reply-To: <01CF8F20-3D1E-4FDE-B4CA-FDE957EC4A56@media.mit.edu> References: <507bbca663d1c5f587b4a1bc0dd749e8.squirrel@mail.stardawn.org> <3B71F448-DC09-45C8-BA1D-E81C45CABEBF@media.mit.edu> <01CF8F20-3D1E-4FDE-B4CA-FDE957EC4A56@media.mit.edu> Message-ID: > How can you guarantee that the postgres cache is coherent across machines? When something is updated invalidation notifications are recorded. Notifications are read and applied before each transaction. > Also, BDB does caching in shared memory regions across multiple processes, > so I'm not sure how much extra benefit you'll get from the Postgres > cache. Good point. One thing is that cache is on the lisp side so there is no overhead associated with traversal/locking/deserialization. From alex.mizrahi at gmail.com Mon May 9 13:17:46 2011 From: alex.mizrahi at gmail.com (Alex Mizrahi) Date: Mon, 09 May 2011 16:17:46 +0300 Subject: [elephant-devel] topo sort? Message-ID: hi I was inspecting code of rebuild-slot-indices and it looks like it has a problem with inherited indices (although it is not easy to reproduce it). rebuild-slot-index drops index first. With inherited indices it tries to drop an index but fails (because it looks for index for this class rather than super class), so index is dropped only once (in the base class which has this inherited index). But (I think) this works correctly only if base class's index is rebuild before subclass index. Otherwise index already populated with some entries will be dropped and so some instances will be lost. E.g. if foo is base class which defines inherited slot/index bar and quux is subclass of foo it works correctly if foo is rebuilt before quux. So I think we need to sort classes (topologically) in known-classes function to make sure that order is correct. Or am I missing something? From alex.mizrahi at gmail.com Mon May 9 18:57:12 2011 From: alex.mizrahi at gmail.com (Alex Mizrahi) Date: Mon, 09 May 2011 21:57:12 +0300 Subject: [elephant-devel] How do I populate an index of a slot that wascreated later with values created before slot became indexed? In-Reply-To: <752E6103-1677-47C8-B775-29530EF80D24@media.mit.edu> References: <852EF08C-2473-42A2-BA82-A2715F1069A7@media.mit.edu> <752E6103-1677-47C8-B775-29530EF80D24@media.mit.edu> Message-ID: > I would recommend a complete code review of the schema implementation as a good first step; > that's one area that often causes problems when making changes to different slot types. Yep, I've just noticed this: I've started investigating why derived indices just stop working from time to time. It turns out that they depend on ensure-schemas being called. But now I have no idea when ensure-schemas is called and what it does. So probably it is easier to start with review of the whole thing... > I'm sure you'll catch design or implementation issues that I missed at the time. I hope so :) From eslick at media.mit.edu Mon May 9 19:24:40 2011 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 9 May 2011 12:24:40 -0700 Subject: [elephant-devel] How do I populate an index of a slot that wascreated later with values created before slot became indexed? In-Reply-To: References: <852EF08C-2473-42A2-BA82-A2715F1069A7@media.mit.edu> <752E6103-1677-47C8-B775-29530EF80D24@media.mit.edu> Message-ID: Alex, This may be overkill, but the metaclass system and related support code would probably benefit from a clean, rethought architecture and possible rewrite of the indexing + schemas + database evolution code. Features were added incrementally over the years in response to specific needs and problems and older code was not always fully integrated with newer features (like schemas, which should have come first but actually were the last major feature added). Moreover, there are a few contracts that are implicit in the code that really should be explicit (e.g. the state of a serialized 'chunk' on disk relative to the current schema, what the 'current' schema is and how it is established, the evolution of the state of an index, etc). Up to you, but I can probably take some time out to review any proposals in early-to-mid June and again in late July. Ian On May 9, 2011, at 11:57 AM, Alex Mizrahi wrote: > >> I would recommend a complete code review of the schema implementation as a good first step; >> that's one area that often causes problems when making changes to different slot types. > > Yep, I've just noticed this: I've started investigating why derived > indices just stop working from time to time. It turns out that they > depend on ensure-schemas being called. But now I have no idea when > ensure-schemas is called and what it does. > So probably it is easier to start with review of the whole thing... > >> I'm sure you'll catch design or implementation issues that I missed > at the time. > > I hope so :) > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From alex.mizrahi at gmail.com Thu May 12 17:05:07 2011 From: alex.mizrahi at gmail.com (Alex Mizrahi) Date: Thu, 12 May 2011 20:05:07 +0300 Subject: [elephant-devel] finalize-inheritance Message-ID: I've found the problem: finalize-inheritance is not always called on class updates (even when compute-slots is called). Spec says that it is called at least once somewhere between class being defined and instantiation of its instances, and that's it. CCL calls it just once for class and never again: Clozure Common Lisp Version 1.6-r14468M (LinuxX8664)! http://paste.lisp.org/display/121960 SBCL calls finalize-inheritance for redefined class (if it is finalized) but not for dependent classes. http://paste.lisp.org/display/121960#2 So I think we cannot depend on finalize-inheritance to detect class redefinitions/updates. What we can depend on, though? I'm not a MOP expert, but it looks like COMPUTE-SLOTS is ideal for slot-related stuff, like derived index triggers. As for schema stuff, I don't know... But it looks like we care mostly about slots, and we don't need layout to be computed. So maybe we can move all class definition/redefinition/update related code to COMPUTE-SLOTS method (after standard method have worked, obviously). This is going to be 100% foolproof, as I don't see how implementation can sidestep COMPUTE-SLOTS when doing anything about class. As for alternatives, I don't know... SHARED-INITIALIZE is not called for updated classes either, so maybe shared-initialize + update-dependent combo or something. From henrik at evahjelte.com Fri May 13 05:18:05 2011 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 13 May 2011 07:18:05 +0200 Subject: [elephant-devel] finalize-inheritance In-Reply-To: References: Message-ID: On Thu, May 12, 2011 at 7:05 PM, Alex Mizrahi wrote: > I've found the problem: finalize-inheritance is not always called on > class updates (even when compute-slots is called). Spec says that it is > called at least once somewhere between class being defined and > instantiation of its instances, and that's it. > What we can depend on, though? I'm not a MOP expert, but it looks like > COMPUTE-SLOTS is ideal for slot-related stuff, like derived index triggers. What about an :after or :before method on update-instance-for-redefined-class? It is the method suggested in "OO programming in Common-Lisp" by Keene. Or look at what is in Closer-MOP, or even use it. Also I found AMOP is online http://keke.yamfa.net/amop/ But maybe your idea works. /Henrik From alex.mizrahi at gmail.com Fri May 13 14:23:59 2011 From: alex.mizrahi at gmail.com (Alex Mizrahi) Date: Fri, 13 May 2011 17:23:59 +0300 Subject: [elephant-devel] finalize-inheritance In-Reply-To: References: Message-ID: > What about an :after or :before method on update-instance-for-redefined-class? It is called once for each instance, but we want update once per class. But we can hook make-instances-obsolete GF which is called once for class. > But maybe your idea works. Hooking compute-slots definitely works for derived slots. (I had to rewrite some code, though.) As for schema stuff, I'm not quite sure what we're trying to achieve, so I need to look into this some more. (E.g. as Ian suggested look through the whole schema thing.) > Or look at what is in Closer-MOP, or even use it. I've asked this question on comp.lang.lisp and got some attention from Pascal Costanza, Closer-MOP's author; so I hope he explains how to do this correctly. From elliottslaughter at gmail.com Sun May 15 21:43:51 2011 From: elliottslaughter at gmail.com (Elliott Slaughter) Date: Sun, 15 May 2011 14:43:51 -0700 Subject: [elephant-devel] Looking for Elephant maintainers Message-ID: By the way, Hans Hubner appears to be looking for the Elephant maintainers (apparently there is a dead email or something): http://netzhansa.blogspot.com/2011/05/orphaned-projects-on-common-lispnet.html -- Elliott Slaughter "Don't worry about what anybody else is going to do. The best way to predict the future is to invent it." - Alan Kay -------------- next part -------------- An HTML attachment was scrubbed... URL: