From read at robertlread.net Tue Sep 13 20:04:24 2005 From: read at robertlread.net (Robert L. Read) Date: Tue, 13 Sep 2005 15:04:24 -0500 Subject: [elephant-devel] Is there any reason not to develop a Postgresql back-end? Message-ID: <1126641864.26213.4.camel@localhost.localdomain> I am considering extending Elephant to allow the use of Postgresql as well as Berkeley-db. This would make the use of Elephant easier to use for those users who are not developing an open-source solution and do not wish to pay for an Berkeley-db license. Can anyone think of a reason I shouldn't undertake this? Is anybody already working on it (or on some other Relational back-end?) Perhaps I am naive, but I expect it to take about two weeks. ---- Robert L. Read, PhD read &T robertlread.net Consider visiting Progressive Engineering: http://robertlread.net/pe In Austin: 912-8593 "Think globally, Act locally." -- RBF -------------- next part -------------- An HTML attachment was scrubbed... URL: From mega at hotpop.com Wed Sep 14 08:20:24 2005 From: mega at hotpop.com (=?utf-8?q?G=C3=A1bor_Melis?=) Date: Wed, 14 Sep 2005 10:20:24 +0200 Subject: [elephant-devel] Is there any reason not to develop a Postgresqlb ack-end? In-Reply-To: <1126641864.26213.4.camel@localhost.localdomain> References: <1126641864.26213.4.camel@localhost.localdomain> Message-ID: <200509141020.24687.mega@hotpop.com> On Tuesday 13 September 2005 22:04, Robert L. Read wrote: > I am considering extending Elephant to allow the use of Postgresql as > well as Berkeley-db. > This would make the use of Elephant easier to use for those users who > are not developing an > open-source solution and do not wish to pay for an Berkeley-db > license. > > Can anyone think of a reason I shouldn't undertake this? Is anybody > already working on it > (or on some other Relational back-end?) > > Perhaps I am naive, but I expect it to take about two weeks. Nobody that I know is working on it. There *may* be some difficulty due to transaction handling semantics (retries/aborts/nesting), but last I checked Postgresql had some pretty nifty versioning scheme that I cannot recall :-). I would very much welcome such an effort, though. Cheers, G?bor Melis From read at robertlread.net Thu Sep 15 14:35:38 2005 From: read at robertlread.net (Robert L. Read) Date: Thu, 15 Sep 2005 09:35:38 -0500 Subject: [elephant-devel] Is there any reason not to develop a Postgresqlb ack-end? In-Reply-To: <9157df230509150249274de76a@mail.gmail.com> References: <1126641864.26213.4.camel@localhost.localdomain> <200509141020.24687.mega@hotpop.com> <9157df230509150249274de76a@mail.gmail.com> Message-ID: <1126794939.26213.56.camel@localhost.localdomain> IMHO: I am using Elephant happily now, and want to go live with a system that uses it, but can't abide by the Berkeley-DB licensing restrictions. After 48 hours considering the problem, you are correct---it will require some major rewriting, at least to be elegant. I can remain code-back-compatible, but the class structure of the controller has to be significantly extended. As to CL-SQL, it is a fabulous system; in fact, Elephant can sit on-top of CL-SQL and thus, more or less easily use the 5 or 6 backend databases that CL-SQL supports easily. My goal would be to seamless support that, as well as the Berekely-db, so that one can simultaneously use different systems. One of the beauties of such as system is that one can migrate from one DB implementation to another almost trivially if one can simultaneously read LISP objects out of Sleepycat and put them into Postgres, for example. CL-SQL also, as you point out, provides a relatively easy way of getting CLOS objects into a database, by allowing one to easily specify a mapping onto SQL-data types for each slot. This is similar to Hibernate in the Java world. Although this is valuable for some applications, I think the Elephant persistent object approach is even more valuable. For the work I've been doing for the last 8 months, it is wonderful to be able to change a class and have it just work with Elephant, and not even have that small additional step that CL-SQL requires of mapping things into SQL data types. This savings in labor is relatively small, but worth fighting for! Although I might chicken-out, my current plan is to continue rewriting Elephant so that it can use both Berkeley-DB and CL-SQL. I will then be morally compelled to release this under the GPL, since Elephant is under the GPL, even if not legally compelled to do so since I am not redistributing the code to anyone. I would rather get some kind of opinion or approval from the existing Elephant owners, if they agree it is worthwhile, than to release it as a fork. It will require some approval because it will be a pretty major rewrite; not just a patch or extension. As to owning Elephant, it seems sensible that I could do it if I manage to complete this non-trivial programming project satisfactorily. However, I am not yet a masterful LISP programmer, and moreover have never owned a public software project and have no idea what that entails. If one of y'all can describe the responsibility to me accurately, then I might consider it. Finally, do we have any idea how many people are using Elephant? If I do this work, will I be the only person using it? If it's not going to help very many people, I could just do the extra work to construct the CL-SQL mappings and save myself a lot of time. On Thu, 2005-09-15 at 18:49 +0900, Ben wrote: > we (the elephant owners) have been silent for far too long, sorry. to > make a long story short -- neither of us have time to own elephant > anymore. we've been debating on the side about how to deal with the > lack of time, and have procrastinated more and more.... > > so we're looking for a new owner for elephant. we'll be happy to > answer what questions we can, but i don't forsee having the time to > deal with this for at least a year (or whenever my dissertation is > done.) i'm really sorry to leave you folks hanging. > > the good news is i think it shouldn't be too hard to deal with the > code base, which as you can see is not too big. the code is also > fairly modular: there's some CLOS magic to make persistent objects, a > lisp datatype to byte stream serializer, and then the berkeley db back > end. plus a bit more here and there. > > to answer the immediate question, i don't think there's any reason why > one couldn't write a postgresql back end, but it might require some > thinking about redesign. elephant assumes that the underlying > database is byte-oriented. i assume this is possible with postgresql > but it doesn't really make much sense to use an SQL database without > using the SQL types. that's just my opinion though. if one wanted to > use SQL data types then maybe one should try out CL-SQL or somesuch? > i dunno. i think there are other possibilities for a back end. > someone on the list a while back was talking about writing a lisp back > end. that would be interesting. i think sqllite might also be a > possibility. > > there's a few patches hanging around in my mail folders (probably on > the list archive) which should be applied. the CVS elephant is close > to a release, it just needs some tagging and regression testing. > > sorry i'm writing from korea and i'm rather jetlagged. but i'll be > back in the states soon. > > any takers? > > Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From midfield at gmail.com Thu Sep 15 22:26:23 2005 From: midfield at gmail.com (Ben) Date: Fri, 16 Sep 2005 07:26:23 +0900 Subject: [elephant-devel] Is there any reason not to develop a Postgresqlb ack-end? In-Reply-To: <9157df230509150249274de76a@mail.gmail.com> References: <1126641864.26213.4.camel@localhost.localdomain> <200509141020.24687.mega@hotpop.com> <9157df230509150249274de76a@mail.gmail.com> Message-ID: <9157df2305091515267b97502d@mail.gmail.com> resending this as i forgot to change my email address..... we (the elephant owners) have been silent for far too long, sorry. to make a long story short -- neither of us have time to own elephant anymore. we've been debating on the side about how to deal with the lack of time, and have procrastinated more and more.... so we're looking for a new owner for elephant. we'll be happy to answer what questions we can, but i don't forsee having the time to deal with this for at least a year (or whenever my dissertation is done.) i'm really sorry to leave you folks hanging. the good news is i think it shouldn't be too hard to deal with the code base, which as you can see is not too big. the code is also fairly modular: there's some CLOS magic to make persistent objects, a lisp datatype to byte stream serializer, and then the berkeley db back end. plus a bit more here and there. to answer the immediate question, i don't think there's any reason why one couldn't write a postgresql back end, but it might require some thinking about redesign. elephant assumes that the underlying database is byte-oriented. i assume this is possible with postgresql but it doesn't really make much sense to use an SQL database without using the SQL types. that's just my opinion though. if one wanted to use SQL data types then maybe one should try out CL-SQL or somesuch? i dunno. i think there are other possibilities for a back end. someone on the list a while back was talking about writing a lisp back end. that would be interesting. i think sqllite might also be a possibility. there's a few patches hanging around in my mail folders (probably on the list archive) which should be applied. the CVS elephant is close to a release, it just needs some tagging and regression testing. sorry i'm writing from korea and i'm rather jetlagged. but i'll be back in the states soon. any takers? Ben On 9/14/05, G?bor Melis wrote: > On Tuesday 13 September 2005 22:04, Robert L. Read wrote: > > I am considering extending Elephant to allow the use of Postgresql as > > well as Berkeley-db. > > This would make the use of Elephant easier to use for those users who > > are not developing an > > open-source solution and do not wish to pay for an Berkeley-db > > license. > > > > Can anyone think of a reason I shouldn't undertake this? Is anybody > > already working on it > > (or on some other Relational back-end?) > > > > Perhaps I am naive, but I expect it to take about two weeks. > > Nobody that I know is working on it. > > There *may* be some difficulty due to transaction handling semantics > (retries/aborts/nesting), but last I checked Postgresql had some pretty > nifty versioning scheme that I cannot recall :-). > > I would very much welcome such an effort, though. > > Cheers, > G?bor Melis > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From mega at hotpop.com Fri Sep 16 08:31:23 2005 From: mega at hotpop.com (=?iso-8859-1?q?G=E1bor_Melis?=) Date: Fri, 16 Sep 2005 10:31:23 +0200 Subject: [elephant-devel] Is there any reason not to develop a Postgre sqlback-end? In-Reply-To: <9157df2305091515267b97502d@mail.gmail.com> References: <9157df2305091515267b97502d@mail.gmail.com> Message-ID: <200509161031.24112.mega@hotpop.com> On Friday 16 September 2005 00:26, Ben wrote: > resending this as i forgot to change my email address..... It's the same as before :-) > so we're looking for a new owner for elephant. we'll be happy to > answer what questions we can, but i don't forsee having the time to > deal with this for at least a year (or whenever my dissertation is > done.) i'm really sorry to leave you folks hanging. Full ownership may be overcommitting myself, but I volunteer as a helper (to have a chance to merge my patches and critique other people's code ;-). G?bor From read at robertlread.net Mon Sep 19 16:21:27 2005 From: read at robertlread.net (Robert L. Read) Date: Mon, 19 Sep 2005 11:21:27 -0500 Subject: [elephant-devel] Suspected bug in sleepycat.lisp file Message-ID: <1127146888.26213.87.camel@localhost.localdomain> While working on implementing Elephant on top of CL-SQL, which led me to do base64 encoding of the serialized lisp objects, I discovered what I suspect to be a (normally) unexercised bug in buffer-read-byte, namely that it currently uses ":char" instead of :unsigned-byte. This bug goes unnoticed because the serializer only puts values less than 128 into the position that it reads with this routine. I discovered it because I am using this routine to read out the whole buffer-stream into a byte-vector for the purpose of base64 encoding so that I can put it into a text field in PostGres safely. If you make this change, all of the tests still run exactly as they did. (defun buffer-read-byte (bs) "Read a byte." (declare (optimize (speed 3) (safety 0)) (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (incf (buffer-stream-position bs)) (deref-array (buffer-stream-buffer bs) '(:array :unsigned-byte) position))) @@ -733,7 +736,33 @@ (type buffer-stream bs)) (let ((position (buffer-stream-position bs))) (incf (buffer-stream-position bs)) - (deref-array (buffer-stream-buffer bs) '(:array :char) position))) + (deref-array (buffer-stream-buffer bs) '(:array :unsigned-byte) position))) -------------- next part -------------- An HTML attachment was scrubbed... URL: From midfield at gmail.com Wed Sep 21 04:41:00 2005 From: midfield at gmail.com (Ben Lee) Date: Tue, 20 Sep 2005 23:41:00 -0500 Subject: [elephant-devel] Suspected bug in sleepycat.lisp file In-Reply-To: <1127146888.26213.87.camel@localhost.localdomain> References: <1127146888.26213.87.camel@localhost.localdomain> Message-ID: <4330E45C.9030008@gmail.com> it looks like you're right. but instead of sending me a patch... who wants the keys to the car? (actually i'm not exactly sure how to hand them over.) B Robert L. Read wrote: > While working on implementing Elephant on top of CL-SQL, which > led me to do base64 encoding of the serialized lisp objects, I discovered > what I suspect to be a (normally) unexercised bug in buffer-read-byte, > namely that it currently uses ":char" instead of :unsigned-byte. > > This bug goes unnoticed because the serializer only puts values less than > 128 into the position that it reads with this routine. I discovered it > because I am using this routine to read out the whole buffer-stream into > a byte-vector for the purpose of base64 encoding so that I can put it into > a text field in PostGres safely. > > If you make this change, all of the tests still run exactly as they did. > > > > > (defun buffer-read-byte (bs) > "Read a byte." > (declare (optimize (speed 3) (safety 0)) > (type buffer-stream bs)) > (let ((position (buffer-stream-position bs))) > (incf (buffer-stream-position bs)) > (deref-array (buffer-stream-buffer bs) '(:array _*:unsigned-byte*_) position))) > > @@ -733,7 +736,33 @@ > (type buffer-stream bs)) > (let ((position (buffer-stream-position bs))) > (incf (buffer-stream-position bs)) > - (deref-array (buffer-stream-buffer bs) '(:array :char) position))) > + (deref-array (buffer-stream-buffer bs) '(:array :unsigned-byte) position))) > > > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From read at robertlread.net Wed Sep 21 17:21:52 2005 From: read at robertlread.net (Robert L. Read) Date: Wed, 21 Sep 2005 12:21:52 -0500 Subject: [elephant-devel] Suspected bug in sleepycat.lisp file In-Reply-To: <4330E45C.9030008@gmail.com> References: <1127146888.26213.87.camel@localhost.localdomain> <4330E45C.9030008@gmail.com> Message-ID: <1127323313.6671.36.camel@localhost.localdomain> Can you outline the responsibilities? In addition to time, there are three reasons why I would not make a good owner: 1) I use only SBCL and plan to use only SBCL for the foreseeable future. 2) I am not a super-great LISP coder. 3) I am modifying Elephant to use CL-SQL as a backend. I plan to "release and own" this, in any case. However, fans of Sleepcat may not want to have the nice simple code of Elephant cluttered up with my SQL-related code. Since I plan to own the CL-SQL based Elephant anyway, the real issue is should that stuff be a fork from Elephant or part of Elephant? And, since I'm only half-way done, people might be really unhappy when the code review it. On Tue, 2005-09-20 at 23:41 -0500, Ben Lee wrote: > it looks like you're right. but instead of sending me a patch... > > who wants the keys to the car? (actually i'm not exactly sure how to > hand them over.) > > B > > Robert L. Read wrote: > > While working on implementing Elephant on top of CL-SQL, which > > led me to do base64 encoding of the serialized lisp objects, I discovered > > what I suspect to be a (normally) unexercised bug in buffer-read-byte, > > namely that it currently uses ":char" instead of :unsigned-byte. > > > > This bug goes unnoticed because the serializer only puts values less than > > 128 into the position that it reads with this routine. I discovered it > > because I am using this routine to read out the whole buffer-stream into > > a byte-vector for the purpose of base64 encoding so that I can put it into > > a text field in PostGres safely. > > > > If you make this change, all of the tests still run exactly as they did. > > > > > > > > > > (defun buffer-read-byte (bs) > > "Read a byte." > > (declare (optimize (speed 3) (safety 0)) > > (type buffer-stream bs)) > > (let ((position (buffer-stream-position bs))) > > (incf (buffer-stream-position bs)) > > (deref-array (buffer-stream-buffer bs) '(:array _*:unsigned-byte*_) position))) > > > > @@ -733,7 +736,33 @@ > > (type buffer-stream bs)) > > (let ((position (buffer-stream-position bs))) > > (incf (buffer-stream-position bs)) > > - (deref-array (buffer-stream-buffer bs) '(:array :char) position))) > > + (deref-array (buffer-stream-buffer bs) '(:array :unsigned-byte) position))) > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel ---- Robert L. Read, PhD read &T robertlread.net Consider visiting Progressive Engineering: http://robertlread.net/pe In Austin: 912-8593 "Think globally, Act locally." -- RBF -------------- next part -------------- An HTML attachment was scrubbed... URL: From read at robertlread.net Fri Sep 30 16:53:47 2005 From: read at robertlread.net (Robert L. Read) Date: Fri, 30 Sep 2005 11:53:47 -0500 Subject: [elephant-devel] Postgres backend... Message-ID: <1128099228.7997.59.camel@localhost.localdomain> I'm not sure who owns Elephant right now, but I am about to complete my postgres-backend extention. (I've got 7 tests out of 90 that are still failing, but should have those knocked out soon.) It took me three weeks and I estimated two; but I'm confident now that within two weeks I can have a complete, testable, well-documented system that let's you seemlessly migrate data between Berkeley DB and Postgres. So the real question is, should this be "fork" or elephant, with a new project name, or should it just be an elephant release? (I assume that the owner will of course want to code-review it before making that decision, but I thought it better to keep everyone informed of what I'm doing. Recall that one nice thing about PostGres is that it's licensing terms for someone building a public website are nicer than those of Berkeley DB. I didn't receive a response to this previous message: In addition to time, there are three reasons why I would not make a good owner: 1) I use only SBCL and plan to use only SBCL for the foreseeable future. 2) I am not a super-great LISP coder. 3) I am modifying Elephant to use CL-SQL as a backend. I plan to "release and own" this, in any case. However, fans of Sleepcat may not want to have the nice simple code of Elephant cluttered up with my SQL-related code. Since I plan to own the CL-SQL based Elephant anyway, the real issue is should that stuff be a fork from Elephant or part of Elephant? And, since I'm only half-way done, people might be really unhappy when the code review it. ---- Robert L. Read, PhD read &T robertlread.net Consider visiting Progressive Engineering: http://robertlread.net/pe In Austin: 912-8593 "Think globally, Act locally." -- RBF -------------- next part -------------- An HTML attachment was scrubbed... URL: From mega at hotpop.com Fri Sep 30 16:10:12 2005 From: mega at hotpop.com (=?utf-8?q?G=C3=A1bor_Melis?=) Date: Fri, 30 Sep 2005 18:10:12 +0200 Subject: [elephant-devel] Postgres backend... In-Reply-To: <1128099228.7997.59.camel@localhost.localdomain> References: <1128099228.7997.59.camel@localhost.localdomain> Message-ID: <200509301810.12482.mega@hotpop.com> On Friday 30 September 2005 18:53, Robert L. Read wrote: > I'm not sure who owns Elephant right now, but I am about to complete > my postgres-backend extention. (I've got 7 tests out of 90 that are > still failing, but should have those knocked out soon.) > > It took me three weeks and I estimated two; but I'm confident now > that within two weeks I can have a complete, testable, > well-documented system that let's you seemlessly migrate data between > Berkeley DB and Postgres. > > So the real question is, should this be "fork" or elephant, with a > new project name, or should it just be an elephant release? (I > assume that the owner will of course want to code-review it before > making that decision, > but I thought it better to keep everyone informed of what I'm doing. > > Recall that one nice thing about PostGres is that it's licensing > terms for someone building a public website are nicer than those of > Berkeley DB. Great stuff. As to whether to fork or not, I'd say let's see the patch and then decide. Hopefully, the differences can be ironed out. Gabor From read at robertlread.net Fri Sep 30 17:34:53 2005 From: read at robertlread.net (Robert L. Read) Date: Fri, 30 Sep 2005 12:34:53 -0500 Subject: [elephant-devel] Postgres backend... In-Reply-To: <200509301810.12482.mega@hotpop.com> References: <1128099228.7997.59.camel@localhost.localdomain> <200509301810.12482.mega@hotpop.com> Message-ID: <1128101693.7997.63.camel@localhost.localdomain> OK, excellent. On Fri, 2005-09-30 at 18:10 +0200, G?bor Melis wrote: > On Friday 30 September 2005 18:53, Robert L. Read wrote: > > I'm not sure who owns Elephant right now, but I am about to complete > > my postgres-backend extention. (I've got 7 tests out of 90 that are > > still failing, but should have those knocked out soon.) > > > > It took me three weeks and I estimated two; but I'm confident now > > that within two weeks I can have a complete, testable, > > well-documented system that let's you seemlessly migrate data between > > Berkeley DB and Postgres. > > > > So the real question is, should this be "fork" or elephant, with a > > new project name, or should it just be an elephant release? (I > > assume that the owner will of course want to code-review it before > > making that decision, > > but I thought it better to keep everyone informed of what I'm doing. > > > > Recall that one nice thing about PostGres is that it's licensing > > terms for someone building a public website are nicer than those of > > Berkeley DB. > > Great stuff. As to whether to fork or not, I'd say let's see the patch > and then decide. Hopefully, the differences can be ironed out. > > Gabor > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: