From =?iso-8859-8?Q?=EE=F6 at common-lisp.net  Wed Aug  1 02:15:44 2007
From: =?iso-8859-8?Q?=EE=F6 at common-lisp.net (=?iso-8859-8?Q?=EE=F6 at common-lisp.net)
Date: Wed, 1 Aug 2007 05:15:44 +0300
Subject: [elephant-devel] =?iso-8859-8?b?+eXl5CDr7CDg4uX45CD57OAg+vn36fLl?=
Message-ID: <4fee11d7c6260f39eb498e31001adf34@delek.com>

An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070801/b31ae052/attachment.html>

From henrik at evahjelte.com  Wed Aug  1 08:23:24 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Wed, 1 Aug 2007 10:23:24 +0200
Subject: [elephant-devel] Fwd: some patches
In-Reply-To: <93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
References: <50e8e4f60707231428n524e337ey1d0a7909bbf9ae6e@mail.gmail.com>
	<50e8e4f60707240050p667670fenf7532a2398597932@mail.gmail.com>
	<BE450228-6501-43B1-8C47-B054E086BD0E@csail.mit.edu>
	<1185900639.4846.229.camel@localhost.localdomain>
	<93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
Message-ID: <50e8e4f60708010123j7d93b962na7502c2478c6ed22@mail.gmail.com>

On 7/31/07, Ian Eslick <eslick at csail.mit.edu> wrote:
> The practical problem that led to the current design of index sorting
> is that we cannot use lisp code to define the sorting function for
> serialized values inside BDB Btrees (same problem I imagine that
> Henrik had with postmodern).

Exactly, the postmodern backend relies on how postgresql sorts things.
And when I changed strings to be saved as blobs instead of strings
in order to overcome a limit on the length of strings, I got a problem
with map-index which relies on the ordering of values. I have reverted
the change now, which means that keys as strings are truncated
if they are longer than 2000 characters.

 Instead, there is a hairy custom C
> procedure that is registered with BDB that parses the serialized
> format so that sorting is done first by type (symbol, string, object,
> pobject, etc) followed by ordering within numeric types, strings and
> symbols.  Everything else is ordered based on the byte ordering of
> its serialized representation.
In postmodern, integers and strings are sorted according to how
postgresql sorts things. Objects are sorted by order of creation, older
first, and equal serialized representation gets the same number.

> To map across indices correctly, we need to know up front whether the
> start value is less than the end value.

Yes now, but in the the common special case where you want to map over
the same value you shouldn't actually need a sorting. You could just
let the backend position the cursor on the value where the key is the
value you want, and move
the cursor over all duplicates. For this case, you need not in theory have an
ordering of the values, only the ability to compare equality.

But this is no longer a big problem, since I reverted how postmodern works.
However if you use objects as keys in btree, relying on the ordering of their
serialized representation might not get what you expect when you query
for identical objects. This problem is shared with the BDB backend.

> Ideally postmodern would have a similar sorting function that
> properly interprets the serialized format just like the BDB function
> does.
It is at least quite similar.

> I think it's best to have a single standard ordering that is as close
> to lisp's notion of ordering as possible so we don't have to maintain
> different orderings.
I agree.

/Henrik


From read at robertlread.net  Thu Aug  2 21:55:04 2007
From: read at robertlread.net (Robert L. Read)
Date: Thu, 02 Aug 2007 16:55:04 -0500
Subject: [elephant-devel] Fwd: some patches
In-Reply-To: <93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
References: <50e8e4f60707231428n524e337ey1d0a7909bbf9ae6e@mail.gmail.com>
	<50e8e4f60707240050p667670fenf7532a2398597932@mail.gmail.com>
	<BE450228-6501-43B1-8C47-B054E086BD0E@csail.mit.edu>
	<1185900639.4846.229.camel@localhost.localdomain>
	<93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
Message-ID: <1186091704.4831.36.camel@localhost.localdomain>

Yes, I think I understand this.  However, a costly alternative does
exist:  just never let
BDB use its own order.  Always impose one that we can compute in lisp.
Then in 
BDB you store a (position,value) pair instead of a value, and either
ensure that BDB 
sorts on the first part of the binary representation of the position the
way you want it to,
or you add a lot of logic into the "cursor-next" operation.  This is how
it is done on the 
CLSQL side.

It is almost certainly a bit slower, and it is certainly a bit harder to
code.

There is a hidden danger in relying upon an order based on the
serialized value---namely,
you can now not swap out serializers without drastic side effects.
Since one of the main
ways that we can improve performance is by writing better serializers
(and, in particular,
serializers that are specific to particular data types), this seems like
a bad idea.

It seems to me that the root of the problem is that BDB does indeed
order based on a 
serialized value.  That is what we should remove.  Certainly, if someone
were to write 
a Pure-LISP backend, which I hope will occur eventually, it would seem
silly for them to 
have to respect an artifact inherited from BDB, when part of the purpose
of such a project
is to escape dependence on BDB.

Forgive me if I'm confused but I assert that we should reverse your
argument: we should
force BDB to be isomorphic to a lisp sorter, not build a lisp sorted
isomorphic to BDB.


On Tue, 2007-07-31 at 15:24 -0400, Ian Eslick wrote:

> The practical problem that led to the current design of index sorting  
> is that we cannot use lisp code to define the sorting function for  
> serialized values inside BDB Btrees (same problem I imagine that  
> Henrik had with postmodern).  Instead, there is a hairy custom C  
> procedure that is registered with BDB that parses the serialized  
> format so that sorting is done first by type (symbol, string, object,  
> pobject, etc) followed by ordering within numeric types, strings and  
> symbols.  Everything else is ordered based on the byte ordering of  
> its serialized representation.
> 
> To map across indices correctly, we need to know up front whether the  
> start value is less than the end value.  And so we need a standard  
> lisp function that is isomorphic to the BDB sorting function.   
> Ideally postmodern would have a similar sorting function that  
> properly interprets the serialized format just like the BDB function  
> does.
> 
> I think it's best to have a single standard ordering that is as close  
> to lisp's notion of ordering as possible so we don't have to maintain  
> different orderings.
> 
> Ian
> 
> PS - It might be possible to have a lisp ordering function implement  
> BDB's notion of sorting by registering it as a callback, however it  
> would have to deserialize the BDB values each time.  So the problems  
> with this are both stability concerns for foreign callbacks and the  
> performance impact of serialization/deserialization for internal BDB  
> operations.  On the cleanliness/performance axis, I think the current  
> approach is the right tradeoff (it's the original one Ben made, FYI).
> 
> On Jul 31, 2007, at 12:50 PM, Robert L. Read wrote:
> 
> > Personally, I think the only sensible way to handle this problem is  
> > to require the user to
> > specify an ordering function.  We can of course provide a default,  
> > which will be error-prone
> > but tend to work most of the time.
> >
> > The function called "my-generic-less-than" which is in the source  
> > tree now could be
> > a starting point for a generic ordering.
> >
> >
> > On Tue, 2007-07-24 at 09:48 -0400, Ian Eslick wrote:
> >> Robert and I have had some extended discussions on ordering in  
> >> indices. I think that all we really need to agree on is _some_  
> >> canonical ordering. If we have mixed types in an index, how should  
> >> they be ordered relative to each other? In BDB we have a C  
> >> function which implements the ordering based on the type tag and  
> >> then based on the type within it. Are you relying on a pure binary  
> >> sort in postmodern? Robert or I will get to submitting that patch  
> >> shortly. I have recently sent in a patch to lisp-compare<= so  
> >> we'll see if we had to make parallel changes. Thanks, Ian On Jul  
> >> 24, 2007, at 3:50 AM, Henrik Hjelte wrote: > I sent this message  
> >> yesterday but I guess it got stuck in the mailing > list filter.  
> >> Perhaps the attachment was too big. Since my > common-lisp.net  
> >> user hhjelte does not have write access to elephant I > have  
> >> placed the patches from here instead: > darcs get http://common- 
> >> lisp.net/project/grand-prix/darcs/elephant > > ----------  
> >> Forwarded message ---------- > From: Henrik Hjelte  
> >> <henrik at evahjelte.com> > Date: Jul 23, 2007 11:28 PM > Subject:  
> >> some patches > To: elephant-devel at common-lisp.net > > > Here are  
> >> some darcs patches that might be of interest. I had some >  
> >> problems with map-index on db-postmodern that made me almost rip  
> >> my > hair of, but finally I made it to work again. The problem is  
> >> that > map-index for a string value rely on the ordering in the  
> >> btree > (continue-p makes use of less than for strings). The  
> >> postmodern > backend relies on how the database backend orders  
> >> things, which is not > always the same thing. Is it a necessary  
> >> feature that b-trees of > string and objects are required to be  
> >> ordered by lisp-compare<=? > > In the process of solving the bug I  
> >> have upgraded the test framework > to use FiveAM instead of RT, It  
> >> has in my opinion a very nice syntax > and some useful features to  
> >> track dependencies between tests. I hope > you agree that it  
> >> improves on things. > > /Henrik Hjelte >  
> >> _______________________________________________ > elephant-devel  
> >> site list > elephant-devel at common-lisp.net > http://common- 
> >> lisp.net/mailman/listinfo/elephant-devel  
> >> _______________________________________________ elephant-devel  
> >> site list elephant-devel at common-lisp.net http://common-lisp.net/ 
> >> mailman/listinfo/elephant-devel
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070802/668c8f50/attachment.html>

From read at robertlread.net  Thu Aug  2 21:55:04 2007
From: read at robertlread.net (Robert L. Read)
Date: Thu, 02 Aug 2007 16:55:04 -0500
Subject: [elephant-devel] Fwd: some patches
In-Reply-To: <93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
References: <50e8e4f60707231428n524e337ey1d0a7909bbf9ae6e@mail.gmail.com>
	<50e8e4f60707240050p667670fenf7532a2398597932@mail.gmail.com>
	<BE450228-6501-43B1-8C47-B054E086BD0E@csail.mit.edu>
	<1185900639.4846.229.camel@localhost.localdomain>
	<93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
Message-ID: <1186091704.4831.36.camel@localhost.localdomain>

Yes, I think I understand this.  However, a costly alternative does
exist:  just never let
BDB use its own order.  Always impose one that we can compute in lisp.
Then in 
BDB you store a (position,value) pair instead of a value, and either
ensure that BDB 
sorts on the first part of the binary representation of the position the
way you want it to,
or you add a lot of logic into the "cursor-next" operation.  This is how
it is done on the 
CLSQL side.

It is almost certainly a bit slower, and it is certainly a bit harder to
code.

There is a hidden danger in relying upon an order based on the
serialized value---namely,
you can now not swap out serializers without drastic side effects.
Since one of the main
ways that we can improve performance is by writing better serializers
(and, in particular,
serializers that are specific to particular data types), this seems like
a bad idea.

It seems to me that the root of the problem is that BDB does indeed
order based on a 
serialized value.  That is what we should remove.  Certainly, if someone
were to write 
a Pure-LISP backend, which I hope will occur eventually, it would seem
silly for them to 
have to respect an artifact inherited from BDB, when part of the purpose
of such a project
is to escape dependence on BDB.

Forgive me if I'm confused but I assert that we should reverse your
argument: we should
force BDB to be isomorphic to a lisp sorter, not build a lisp sorted
isomorphic to BDB.


On Tue, 2007-07-31 at 15:24 -0400, Ian Eslick wrote:

> The practical problem that led to the current design of index sorting  
> is that we cannot use lisp code to define the sorting function for  
> serialized values inside BDB Btrees (same problem I imagine that  
> Henrik had with postmodern).  Instead, there is a hairy custom C  
> procedure that is registered with BDB that parses the serialized  
> format so that sorting is done first by type (symbol, string, object,  
> pobject, etc) followed by ordering within numeric types, strings and  
> symbols.  Everything else is ordered based on the byte ordering of  
> its serialized representation.
> 
> To map across indices correctly, we need to know up front whether the  
> start value is less than the end value.  And so we need a standard  
> lisp function that is isomorphic to the BDB sorting function.   
> Ideally postmodern would have a similar sorting function that  
> properly interprets the serialized format just like the BDB function  
> does.
> 
> I think it's best to have a single standard ordering that is as close  
> to lisp's notion of ordering as possible so we don't have to maintain  
> different orderings.
> 
> Ian
> 
> PS - It might be possible to have a lisp ordering function implement  
> BDB's notion of sorting by registering it as a callback, however it  
> would have to deserialize the BDB values each time.  So the problems  
> with this are both stability concerns for foreign callbacks and the  
> performance impact of serialization/deserialization for internal BDB  
> operations.  On the cleanliness/performance axis, I think the current  
> approach is the right tradeoff (it's the original one Ben made, FYI).
> 
> On Jul 31, 2007, at 12:50 PM, Robert L. Read wrote:
> 
> > Personally, I think the only sensible way to handle this problem is  
> > to require the user to
> > specify an ordering function.  We can of course provide a default,  
> > which will be error-prone
> > but tend to work most of the time.
> >
> > The function called "my-generic-less-than" which is in the source  
> > tree now could be
> > a starting point for a generic ordering.
> >
> >
> > On Tue, 2007-07-24 at 09:48 -0400, Ian Eslick wrote:
> >> Robert and I have had some extended discussions on ordering in  
> >> indices. I think that all we really need to agree on is _some_  
> >> canonical ordering. If we have mixed types in an index, how should  
> >> they be ordered relative to each other? In BDB we have a C  
> >> function which implements the ordering based on the type tag and  
> >> then based on the type within it. Are you relying on a pure binary  
> >> sort in postmodern? Robert or I will get to submitting that patch  
> >> shortly. I have recently sent in a patch to lisp-compare<= so  
> >> we'll see if we had to make parallel changes. Thanks, Ian On Jul  
> >> 24, 2007, at 3:50 AM, Henrik Hjelte wrote: > I sent this message  
> >> yesterday but I guess it got stuck in the mailing > list filter.  
> >> Perhaps the attachment was too big. Since my > common-lisp.net  
> >> user hhjelte does not have write access to elephant I > have  
> >> placed the patches from here instead: > darcs get http://common- 
> >> lisp.net/project/grand-prix/darcs/elephant > > ----------  
> >> Forwarded message ---------- > From: Henrik Hjelte  
> >> <henrik at evahjelte.com> > Date: Jul 23, 2007 11:28 PM > Subject:  
> >> some patches > To: elephant-devel at common-lisp.net > > > Here are  
> >> some darcs patches that might be of interest. I had some >  
> >> problems with map-index on db-postmodern that made me almost rip  
> >> my > hair of, but finally I made it to work again. The problem is  
> >> that > map-index for a string value rely on the ordering in the  
> >> btree > (continue-p makes use of less than for strings). The  
> >> postmodern > backend relies on how the database backend orders  
> >> things, which is not > always the same thing. Is it a necessary  
> >> feature that b-trees of > string and objects are required to be  
> >> ordered by lisp-compare<=? > > In the process of solving the bug I  
> >> have upgraded the test framework > to use FiveAM instead of RT, It  
> >> has in my opinion a very nice syntax > and some useful features to  
> >> track dependencies between tests. I hope > you agree that it  
> >> improves on things. > > /Henrik Hjelte >  
> >> _______________________________________________ > elephant-devel  
> >> site list > elephant-devel at common-lisp.net > http://common- 
> >> lisp.net/mailman/listinfo/elephant-devel  
> >> _______________________________________________ elephant-devel  
> >> site list elephant-devel at common-lisp.net http://common-lisp.net/ 
> >> mailman/listinfo/elephant-devel
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070802/668c8f50/attachment-0001.html>

From ml13 at onlinehome.de  Fri Aug  3 13:48:40 2007
From: ml13 at onlinehome.de (Kilian Sprotte)
Date: Fri, 3 Aug 2007 15:48:40 +0200
Subject: [elephant-devel] trivial patch
Message-ID: <274173BE-2C3B-4757-B0DE-507E0D5EFECA@onlinehome.de>

Hello,

I have a very simple patch:

controller-lost-error should inherit from simple-error, I think.

(define-condition controller-lost-error (simple-error)
    ((object :initarg :object :accessor store-controller-closed-error- 
object)
     (spec :initarg :spec :accessor store-controller-closed-error- 
spec)))

(defun signal-controller-lost-error (object)
   (cerror "Open a new instance and continue?"
	  'controller-lost-error
	  :format-control "Store controller for specification ~A for object  
~A cannot be found."
	  :format-arguments (list object (dbcn-spc-pst object))
	  :object object
	  :spec (dbcn-spc-pst object)))

I am amazed, how elephant has developed - wow!!

Best,
   Kilian


-------------- next part --------------
A non-text attachment was scrubbed...
Name: controller-lost-error.patch
Type: application/octet-stream
Size: 4503 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070803/a4612d7f/attachment.obj>

From eslick at csail.mit.edu  Fri Aug  3 14:01:18 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Fri, 3 Aug 2007 10:01:18 -0400
Subject: [elephant-devel] Fwd: some patches
In-Reply-To: <1186091704.4831.36.camel@localhost.localdomain>
References: <50e8e4f60707231428n524e337ey1d0a7909bbf9ae6e@mail.gmail.com>
	<50e8e4f60707240050p667670fenf7532a2398597932@mail.gmail.com>
	<BE450228-6501-43B1-8C47-B054E086BD0E@csail.mit.edu>
	<1185900639.4846.229.camel@localhost.localdomain>
	<93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
	<1186091704.4831.36.camel@localhost.localdomain>
Message-ID: <C3427D1C-D0A0-4D49-955F-4A0F9E776801@csail.mit.edu>

First of all, BDB does not sort on the serialized values except for  
values for which no lisp ordering exists.  BDB is given a C function  
which decodes the serialized format on the fly, without talking to  
Lisp, but properly orders all of lisp's orderable objects.

Orderable lisp types:
1. Numeric types: all numeric types are sorted identically to lisp (=  
< > <= >=, etc)
    Numeric types include fixnum, integer, float, double-float, rational
2. String types: all string types are sorted identically to lisp  
(string< string<= string=, etc)

Unorderable lisp types:
1. Symbols: I do sort symbols by symbol-name as a convenience.
    Lisp does not provide an ordering fn for symbols so this is additive
2. Objects, structs, etc: lisp does not provide an ordering function  
for objects
3. Arrays, hash tables, etc: lisp does not provide an ordering  
function for these types

Some of these types can be grouped by eq, eql, equal or equalp.  More  
on this below.

Because a BTree requires some order for all objects, we need to  
expand on the lisp specification of ordering.  First we sort on type  
so that all unorderable objects of a given type are stored as a  
group.  Then we choose an arbitrary order (binary rep).  Now most of  
the hairy issues arise when we mix types in an index and want to  
traverse a range in the index or do cursor ops on an index.  The  
specification is this:

1) Elephant makes no guarantee about the order of unorderable types.   
Portable implementations should not depend on an order being  
consistent.  Unorderable types should be treated as sets.  Anything  
that is equalp in lisp will be grouped together in the data store  
(arrays, structs, symbols)

2) Elephant makes no guarantee about the relative order of different  
types in an index.

*** There is an open issue: how to specify a range by object type.   
You can't do :start and :end because you don't know what object will  
be at the start and end due to #1 and #2 above.  I think this should  
be dealt with as a special case and a less efficient procedure use

I can't agree with either of your proposals due to performance issues:

1) The performance impact of (position, value) is extreme.  I turn  
all of the BTree O(log N) operations into O(N).  I now have to read  
all values out of the BTree until I identify the insertion point,  
then renumber all the following elements.  This defeats the whole  
purpose of having a BTree in the first place.

2) I hate to say it but the CL-SQL approach to cursor operations is  
not just terribly inefficient, it also defeats the purpose of having  
indicies on disk by requiring that all values be pulled into memory  
to support cursor or map operations.  I see the CL-SQL backend as a  
solution to licensing problems for small databases.  For complex  
queries over large databases one is better off with an object  
relational solution like that implemented by CL-SQL's native metaclass.

The reason that lisp needs to implement a more complex sorting  
function to support cursor operations over BTrees is that BTrees  
benefit from ordering types relative to each other (we can't  
intersperse numeric and non-numeric types without violating lisp  
ordering) and the cursor fn's need to mirror this.  This doesn't  
violate lisp ordering, it is additive.  The clarity of the required  
constraints could be improved, though.

 > There is a hidden danger in relying upon an order based on the  
serialized value---namely,
 > you can now not swap out serializers without drastic side  
effects.  Since one of the main
 > ways that we can improve performance is by writing better  
serializers (and, in particular,
 > serializers that are specific to particular data types), this  
seems like a bad idea.

If designers adhere to the two restrictions above, then a new  
serializer that supports lisp ordering and introduces a new ordering  
of types or within types that are unordered in lisp will have no  
impact on application code.

Sorry if any point is unclear, I'm in a hurry.  Please ask clarifying  
questions if necessary.

Ian

PS - This leaves me with a question.  Is it possible in any  
relational DB to register a sorting fn that you can sort by when  
doing a sorted query?  Typically rows are sorted by their unique ID  
field (i.e. the key used in the underlying BTree).



On Aug 2, 2007, at 5:55 PM, Robert L. Read wrote:

> Yes, I think I understand this.  However, a costly alternative does  
> exist:  just never let
> BDB use its own order.  Always impose one that we can compute in  
> lisp.  Then in
> BDB you store a (position,value) pair instead of a value, and  
> either ensure that BDB
> sorts on the first part of the binary representation of the  
> position the way you want it to,
> or you add a lot of logic into the "cursor-next" operation.  This  
> is how it is done on the
> CLSQL side.
>
> It is almost certainly a bit slower, and it is certainly a bit  
> harder to code.
>
> It seems to me that the root of the problem is that BDB does indeed  
> order based on a
> serialized value.  That is what we should remove.  Certainly, if  
> someone were to write
> a Pure-LISP backend, which I hope will occur eventually, it would  
> seem silly for them to
> have to respect an artifact inherited from BDB, when part of the  
> purpose of such a project
> is to escape dependence on BDB.
>
> Forgive me if I'm confused but I assert that we should reverse your  
> argument: we should
> force BDB to be isomorphic to a lisp sorter, not build a lisp  
> sorted isomorphic to BDB.
>
>
> On Tue, 2007-07-31 at 15:24 -0400, Ian Eslick wrote:
>> The practical problem that led to the current design of index  
>> sorting is that we cannot use lisp code to define the sorting  
>> function for serialized values inside BDB Btrees (same problem I  
>> imagine that Henrik had with postmodern). Instead, there is a  
>> hairy custom C procedure that is registered with BDB that parses  
>> the serialized format so that sorting is done first by type  
>> (symbol, string, object, pobject, etc) followed by ordering within  
>> numeric types, strings and symbols. Everything else is ordered  
>> based on the byte ordering of its serialized representation. To  
>> map across indices correctly, we need to know up front whether the  
>> start value is less than the end value. And so we need a standard  
>> lisp function that is isomorphic to the BDB sorting function.  
>> Ideally postmodern would have a similar sorting function that    
>> properly interprets the serialized format just like the BDB  
>> function does. I think it's best to have a single standard  
>> ordering that is as close to lisp's notion of ordering as possible  
>> so we don't have to maintain different orderings. Ian PS - It  
>> might be possible to have a lisp ordering function implement BDB's  
>> notion of sorting by registering it as a callback, however it  
>> would have to deserialize the BDB values each time. So the  
>> problems with this are both stability concerns for foreign  
>> callbacks and the performance impact of serialization/ 
>> deserialization for internal BDB operations. On the cleanliness/ 
>> performance axis, I think the current approach is the right  
>> tradeoff (it's the original one Ben made, FYI). On Jul 31, 2007,  
>> at 12:50 PM, Robert L. Read wrote: > Personally, I think the only  
>> sensible way to handle this problem is > to require the user to >  
>> specify an ordering function. We can of course provide a default,  
>> > which will be error-prone > but tend to work most of the time. >  
>> > The function called "my-generic-less-than" which is in the  
>> source > tree now could be > a starting point for a generic  
>> ordering. > > > On Tue, 2007-07-24 at 09:48 -0400, Ian Eslick  
>> wrote: >> Robert and I have had some extended discussions on  
>> ordering in >> indices. I think that all we really need to agree  
>> on is _some_ >> canonical ordering. If we have mixed types in an  
>> index, how should >> they be ordered relative to each other? In  
>> BDB we have a C >> function which implements the ordering based on  
>> the type tag and >> then based on the type within it. Are you  
>> relying on a pure binary >> sort in postmodern? Robert or I will  
>> get to submitting that patch >> shortly. I have recently sent in a  
>> patch to lisp-compare<= so >> we'll see if we had to make parallel  
>> changes. Thanks, Ian On Jul >> 24, 2007, at 3:50 AM, Henrik Hjelte  
>> wrote: > I sent this message >> yesterday but I guess it got stuck  
>> in the mailing > list filter. >> Perhaps the attachment was too  
>> big. Since my > common-lisp.net >> user hhjelte does not have  
>> write access to elephant I > have >> placed the patches from here  
>> instead: > darcs get http://common- >> lisp.net/project/grand-prix/ 
>> darcs/elephant > > ---------- >> Forwarded message ---------- >  
>> From: Henrik Hjelte >> <henrik at evahjelte.com> > Date: Jul 23, 2007  
>> 11:28 PM > Subject: >> some patches > To: elephant-devel at common- 
>> lisp.net > > > Here are >> some darcs patches that might be of  
>> interest. I had some > >> problems with map-index on db-postmodern  
>> that made me almost rip >> my > hair of, but finally I made it to  
>> work again. The problem is >> that > map-index for a string value  
>> rely on the ordering in the >> btree > (continue-p makes use of  
>> less than for strings). The >> postmodern > backend relies on how  
>> the database backend orders >> things, which is not > always the  
>> same thing. Is it a necessary >> feature that b-trees of > string  
>> and objects are required to be >> ordered by lisp-compare<=? > >  
>> In the process of solving the bug I >> have upgraded the test  
>> framework > to use FiveAM instead of RT, It >> has in my opinion a  
>> very nice syntax > and some useful features to >> track  
>> dependencies between tests. I hope > you agree that it >> improves  
>> on things. > > /Henrik Hjelte > >>  
>> _______________________________________________ > elephant-devel  
>> >> site list > elephant-devel at common-lisp.net > http://common- >>  
>> lisp.net/mailman/listinfo/elephant-devel >>  
>> _______________________________________________ elephant-devel >>  
>> site list elephant-devel at common-lisp.net http://common-lisp.net/  
>> >> mailman/listinfo/elephant-devel >  
>> _______________________________________________ > elephant-devel  
>> site list > elephant-devel at common-lisp.net > http://common- 
>> lisp.net/mailman/listinfo/elephant-devel  
>> _______________________________________________ elephant-devel  
>> site list elephant-devel at common-lisp.net http://common-lisp.net/ 
>> mailman/listinfo/elephant-devel
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Fri Aug  3 14:01:18 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Fri, 3 Aug 2007 10:01:18 -0400
Subject: [elephant-devel] Fwd: some patches
In-Reply-To: <1186091704.4831.36.camel@localhost.localdomain>
References: <50e8e4f60707231428n524e337ey1d0a7909bbf9ae6e@mail.gmail.com>
	<50e8e4f60707240050p667670fenf7532a2398597932@mail.gmail.com>
	<BE450228-6501-43B1-8C47-B054E086BD0E@csail.mit.edu>
	<1185900639.4846.229.camel@localhost.localdomain>
	<93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
	<1186091704.4831.36.camel@localhost.localdomain>
Message-ID: <C3427D1C-D0A0-4D49-955F-4A0F9E776801@csail.mit.edu>

First of all, BDB does not sort on the serialized values except for  
values for which no lisp ordering exists.  BDB is given a C function  
which decodes the serialized format on the fly, without talking to  
Lisp, but properly orders all of lisp's orderable objects.

Orderable lisp types:
1. Numeric types: all numeric types are sorted identically to lisp (=  
< > <= >=, etc)
    Numeric types include fixnum, integer, float, double-float, rational
2. String types: all string types are sorted identically to lisp  
(string< string<= string=, etc)

Unorderable lisp types:
1. Symbols: I do sort symbols by symbol-name as a convenience.
    Lisp does not provide an ordering fn for symbols so this is additive
2. Objects, structs, etc: lisp does not provide an ordering function  
for objects
3. Arrays, hash tables, etc: lisp does not provide an ordering  
function for these types

Some of these types can be grouped by eq, eql, equal or equalp.  More  
on this below.

Because a BTree requires some order for all objects, we need to  
expand on the lisp specification of ordering.  First we sort on type  
so that all unorderable objects of a given type are stored as a  
group.  Then we choose an arbitrary order (binary rep).  Now most of  
the hairy issues arise when we mix types in an index and want to  
traverse a range in the index or do cursor ops on an index.  The  
specification is this:

1) Elephant makes no guarantee about the order of unorderable types.   
Portable implementations should not depend on an order being  
consistent.  Unorderable types should be treated as sets.  Anything  
that is equalp in lisp will be grouped together in the data store  
(arrays, structs, symbols)

2) Elephant makes no guarantee about the relative order of different  
types in an index.

*** There is an open issue: how to specify a range by object type.   
You can't do :start and :end because you don't know what object will  
be at the start and end due to #1 and #2 above.  I think this should  
be dealt with as a special case and a less efficient procedure use

I can't agree with either of your proposals due to performance issues:

1) The performance impact of (position, value) is extreme.  I turn  
all of the BTree O(log N) operations into O(N).  I now have to read  
all values out of the BTree until I identify the insertion point,  
then renumber all the following elements.  This defeats the whole  
purpose of having a BTree in the first place.

2) I hate to say it but the CL-SQL approach to cursor operations is  
not just terribly inefficient, it also defeats the purpose of having  
indicies on disk by requiring that all values be pulled into memory  
to support cursor or map operations.  I see the CL-SQL backend as a  
solution to licensing problems for small databases.  For complex  
queries over large databases one is better off with an object  
relational solution like that implemented by CL-SQL's native metaclass.

The reason that lisp needs to implement a more complex sorting  
function to support cursor operations over BTrees is that BTrees  
benefit from ordering types relative to each other (we can't  
intersperse numeric and non-numeric types without violating lisp  
ordering) and the cursor fn's need to mirror this.  This doesn't  
violate lisp ordering, it is additive.  The clarity of the required  
constraints could be improved, though.

 > There is a hidden danger in relying upon an order based on the  
serialized value---namely,
 > you can now not swap out serializers without drastic side  
effects.  Since one of the main
 > ways that we can improve performance is by writing better  
serializers (and, in particular,
 > serializers that are specific to particular data types), this  
seems like a bad idea.

If designers adhere to the two restrictions above, then a new  
serializer that supports lisp ordering and introduces a new ordering  
of types or within types that are unordered in lisp will have no  
impact on application code.

Sorry if any point is unclear, I'm in a hurry.  Please ask clarifying  
questions if necessary.

Ian

PS - This leaves me with a question.  Is it possible in any  
relational DB to register a sorting fn that you can sort by when  
doing a sorted query?  Typically rows are sorted by their unique ID  
field (i.e. the key used in the underlying BTree).



On Aug 2, 2007, at 5:55 PM, Robert L. Read wrote:

> Yes, I think I understand this.  However, a costly alternative does  
> exist:  just never let
> BDB use its own order.  Always impose one that we can compute in  
> lisp.  Then in
> BDB you store a (position,value) pair instead of a value, and  
> either ensure that BDB
> sorts on the first part of the binary representation of the  
> position the way you want it to,
> or you add a lot of logic into the "cursor-next" operation.  This  
> is how it is done on the
> CLSQL side.
>
> It is almost certainly a bit slower, and it is certainly a bit  
> harder to code.
>
> It seems to me that the root of the problem is that BDB does indeed  
> order based on a
> serialized value.  That is what we should remove.  Certainly, if  
> someone were to write
> a Pure-LISP backend, which I hope will occur eventually, it would  
> seem silly for them to
> have to respect an artifact inherited from BDB, when part of the  
> purpose of such a project
> is to escape dependence on BDB.
>
> Forgive me if I'm confused but I assert that we should reverse your  
> argument: we should
> force BDB to be isomorphic to a lisp sorter, not build a lisp  
> sorted isomorphic to BDB.
>
>
> On Tue, 2007-07-31 at 15:24 -0400, Ian Eslick wrote:
>> The practical problem that led to the current design of index  
>> sorting is that we cannot use lisp code to define the sorting  
>> function for serialized values inside BDB Btrees (same problem I  
>> imagine that Henrik had with postmodern). Instead, there is a  
>> hairy custom C procedure that is registered with BDB that parses  
>> the serialized format so that sorting is done first by type  
>> (symbol, string, object, pobject, etc) followed by ordering within  
>> numeric types, strings and symbols. Everything else is ordered  
>> based on the byte ordering of its serialized representation. To  
>> map across indices correctly, we need to know up front whether the  
>> start value is less than the end value. And so we need a standard  
>> lisp function that is isomorphic to the BDB sorting function.  
>> Ideally postmodern would have a similar sorting function that    
>> properly interprets the serialized format just like the BDB  
>> function does. I think it's best to have a single standard  
>> ordering that is as close to lisp's notion of ordering as possible  
>> so we don't have to maintain different orderings. Ian PS - It  
>> might be possible to have a lisp ordering function implement BDB's  
>> notion of sorting by registering it as a callback, however it  
>> would have to deserialize the BDB values each time. So the  
>> problems with this are both stability concerns for foreign  
>> callbacks and the performance impact of serialization/ 
>> deserialization for internal BDB operations. On the cleanliness/ 
>> performance axis, I think the current approach is the right  
>> tradeoff (it's the original one Ben made, FYI). On Jul 31, 2007,  
>> at 12:50 PM, Robert L. Read wrote: > Personally, I think the only  
>> sensible way to handle this problem is > to require the user to >  
>> specify an ordering function. We can of course provide a default,  
>> > which will be error-prone > but tend to work most of the time. >  
>> > The function called "my-generic-less-than" which is in the  
>> source > tree now could be > a starting point for a generic  
>> ordering. > > > On Tue, 2007-07-24 at 09:48 -0400, Ian Eslick  
>> wrote: >> Robert and I have had some extended discussions on  
>> ordering in >> indices. I think that all we really need to agree  
>> on is _some_ >> canonical ordering. If we have mixed types in an  
>> index, how should >> they be ordered relative to each other? In  
>> BDB we have a C >> function which implements the ordering based on  
>> the type tag and >> then based on the type within it. Are you  
>> relying on a pure binary >> sort in postmodern? Robert or I will  
>> get to submitting that patch >> shortly. I have recently sent in a  
>> patch to lisp-compare<= so >> we'll see if we had to make parallel  
>> changes. Thanks, Ian On Jul >> 24, 2007, at 3:50 AM, Henrik Hjelte  
>> wrote: > I sent this message >> yesterday but I guess it got stuck  
>> in the mailing > list filter. >> Perhaps the attachment was too  
>> big. Since my > common-lisp.net >> user hhjelte does not have  
>> write access to elephant I > have >> placed the patches from here  
>> instead: > darcs get http://common- >> lisp.net/project/grand-prix/ 
>> darcs/elephant > > ---------- >> Forwarded message ---------- >  
>> From: Henrik Hjelte >> <henrik at evahjelte.com> > Date: Jul 23, 2007  
>> 11:28 PM > Subject: >> some patches > To: elephant-devel at common- 
>> lisp.net > > > Here are >> some darcs patches that might be of  
>> interest. I had some > >> problems with map-index on db-postmodern  
>> that made me almost rip >> my > hair of, but finally I made it to  
>> work again. The problem is >> that > map-index for a string value  
>> rely on the ordering in the >> btree > (continue-p makes use of  
>> less than for strings). The >> postmodern > backend relies on how  
>> the database backend orders >> things, which is not > always the  
>> same thing. Is it a necessary >> feature that b-trees of > string  
>> and objects are required to be >> ordered by lisp-compare<=? > >  
>> In the process of solving the bug I >> have upgraded the test  
>> framework > to use FiveAM instead of RT, It >> has in my opinion a  
>> very nice syntax > and some useful features to >> track  
>> dependencies between tests. I hope > you agree that it >> improves  
>> on things. > > /Henrik Hjelte > >>  
>> _______________________________________________ > elephant-devel  
>> >> site list > elephant-devel at common-lisp.net > http://common- >>  
>> lisp.net/mailman/listinfo/elephant-devel >>  
>> _______________________________________________ elephant-devel >>  
>> site list elephant-devel at common-lisp.net http://common-lisp.net/  
>> >> mailman/listinfo/elephant-devel >  
>> _______________________________________________ > elephant-devel  
>> site list > elephant-devel at common-lisp.net > http://common- 
>> lisp.net/mailman/listinfo/elephant-devel  
>> _______________________________________________ elephant-devel  
>> site list elephant-devel at common-lisp.net http://common-lisp.net/ 
>> mailman/listinfo/elephant-devel
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From read at robertlread.net  Fri Aug  3 15:49:59 2007
From: read at robertlread.net (Robert L. Read)
Date: Fri, 03 Aug 2007 10:49:59 -0500
Subject: [elephant-devel] Fwd: some patches
In-Reply-To: <C3427D1C-D0A0-4D49-955F-4A0F9E776801@csail.mit.edu>
References: <50e8e4f60707231428n524e337ey1d0a7909bbf9ae6e@mail.gmail.com>
	<50e8e4f60707240050p667670fenf7532a2398597932@mail.gmail.com>
	<BE450228-6501-43B1-8C47-B054E086BD0E@csail.mit.edu>
	<1185900639.4846.229.camel@localhost.localdomain>
	<93791CD1-7CAF-4B91-B9F8-077D5D148BA9@csail.mit.edu>
	<1186091704.4831.36.camel@localhost.localdomain>
	<C3427D1C-D0A0-4D49-955F-4A0F9E776801@csail.mit.edu>
Message-ID: <1186156200.5467.45.camel@localhost.localdomain>

I agree with all of your last email.

However, I think in your discussion of complexity of the existing
operations in the CL-SQL side
you are implying there are lots of O(n) operations when in fact this
only occurs when one 
indexes a slot that has a small number of values relative to the number
of objects or uses 
a very non-selective functional index.  I wrote that stuff before you
implemented slot indexing.
However, you are correct --- in some cases it will perform very poorly.
However, we now have
the Postmodern back-end (which I am already using in production), and it
probably doesn't suffer
from the same problems.  Moreover, the licensing issues with BDB seem to
have gone away.

I understand that you and I have slightly different approaches.  I am
thinking of Prevalence-style
implementations, that don't rely on disk-based indices (for example, DCM
uses hash-tables), and 
you are thinking of databases way to large to fit into memory, and
relying upon designing an 
index structure using the powerful Elephant mechanism to have an
efficient system.  I much
prefer that than supporting the notion of using an ORM, as you mention.
In fact, I hope Elephant
will be an ORM-zilla --- I hope it will teach people there is no need to
deal with that impedance
mismatch.

I think you clarification of the specification (by saying it is
unspecified) is very helpful.

With respect to the "open issue" you raise about "get-instances-by-
range", I wrote the following 
little article:

I take as a basic assumption that a B-Tree is a data structure, not an
abstract data type.  The abstract data type that it implements is an 
Ordered Set, that is a set equipped with a total order (a total order
being one that is antisymmetic, transitive and complete.)

A system like Elephant, or a relational database, should present an
abstract data type, and not expose implementation details.  That's
we call relational databases "relational" rather than "ISAM-al" or 
"BTree-al".  Even though in practice they don't (since typically
duplicate tuples are allowed, though implementing multisets rather than
sets, properly), what we call RDBMS proper are based upon the notion 
of the relation, an abstract data type.  I think Elephant should 
be based on the simpler notion of a Set or a Dictionary, and 
support the notion of an Ordered Set.

There are two basic Abstract Data Types that Elephant provides. This
is a kind legacy of the original API.  We implement the "Map" ADT and
the "Ordered Set" ADT.  Example of "Map" are "add-to-store(key,value)"
and "get-from-store(key,value)".  Persistent classes are examples
of the "Set" or "Ordered Set" ADT.  A fundamental problem with 
thinking in terms of these Abstract Data Types as defined by the 
Wikipedia, for example, is that they are not defined to have an
"enumerate all values" operation.  This is an abstract operation but
is essential in practice.  If you can't do that, you have to have 
a separate store to keep track of the objects in the store, and that 
is obviously no good.

(We provide nice mechanisms for enumerating all objects in both cases.
We have the "map-btree" function and you have written "map-class".
In practice, there are also cursors to do similar things.)

It seems to me an open question whether we should define persistent
classes to be a "Set" or an "Ordered Set".

As you point out, lisp doesn't provide an ordering for objects.  This
is because, unlike the integers, it is unclear what the ordering should
be.  Even for strings, the ordering should technically be dependent on
the language in which you interpret the strings---not every culture 
orders their letters in the same way.

It seems to me that everything would be a lot cleaner if we think
of fundamentally presenting the "Set" ADT rather than the "Ordered Set"
ADT.  The "Set" ADT does not support "get-instances-by-range", but the 
"Ordered Set" does (however, this must be based on a LISP-defined
order.)

In your previous email you suggest a specification wherein Elephant
makes not guarantee about the order of unorderable types.  This is 
effectively the specification of a "Set" type.  The clearest thing
of course would be our test suite reflected the distinction between
Sets and Ordered Sets by testing the persistence of order on Ordered
Sets but not relying upon it all for Sets.

If one thinks about the asymptotic complexity of both of these kinds
of ADTs, it is reasonable to expect O(log(n)) insertion, deletion, and
lookups in all cases.  

Elephant provides several index mechanisms.  The most elegant is 
the indexing of a slot value which you implemented; but there are 
also function indexes.  For example, one can implement a boolean
function "rhymes-with-cat" and build an index on that.  This allows
you to efficiently enumerate only lisp objects that for which 
"rhymes-with-cat" is true.  More reaonably, one could build an
index on a class slot that had a boolean value.  
This is the place where the current CL-SQL backend has an O(n)
operation--
when there are duplicate keys and index.  Recall that I coded that 
before you wrote the class indexing stuff.  I personally don't think it 
is very important, and I use DCM anyway.  If anybody actually
encounters 
this as a performance problem, I'll be happy to work on it --- 
unless of course you're using
Postgres, in which case I would recommend switching to the new (and not
officially released) Postmodern back-end.  I have done that on my
production site already.



On Fri, 2007-08-03 at 10:01 -0400, Ian Eslick wrote:

> First of all, BDB does not sort on the serialized values except for  
> values for which no lisp ordering exists.  BDB is given a C function  
> which decodes the serialized format on the fly, without talking to  
> Lisp, but properly orders all of lisp's orderable objects.
> 
> Orderable lisp types:
> 1. Numeric types: all numeric types are sorted identically to lisp (=  
> < > <= >=, etc)
>     Numeric types include fixnum, integer, float, double-float, rational
> 2. String types: all string types are sorted identically to lisp  
> (string< string<= string=, etc)
> 
> Unorderable lisp types:
> 1. Symbols: I do sort symbols by symbol-name as a convenience.
>     Lisp does not provide an ordering fn for symbols so this is additive
> 2. Objects, structs, etc: lisp does not provide an ordering function  
> for objects
> 3. Arrays, hash tables, etc: lisp does not provide an ordering  
> function for these types
> 
> Some of these types can be grouped by eq, eql, equal or equalp.  More  
> on this below.
> 
> Because a BTree requires some order for all objects, we need to  
> expand on the lisp specification of ordering.  First we sort on type  
> so that all unorderable objects of a given type are stored as a  
> group.  Then we choose an arbitrary order (binary rep).  Now most of  
> the hairy issues arise when we mix types in an index and want to  
> traverse a range in the index or do cursor ops on an index.  The  
> specification is this:
> 
> 1) Elephant makes no guarantee about the order of unorderable types.   
> Portable implementations should not depend on an order being  
> consistent.  Unorderable types should be treated as sets.  Anything  
> that is equalp in lisp will be grouped together in the data store  
> (arrays, structs, symbols)
> 
> 2) Elephant makes no guarantee about the relative order of different  
> types in an index.
> 
> *** There is an open issue: how to specify a range by object type.   
> You can't do :start and :end because you don't know what object will  
> be at the start and end due to #1 and #2 above.  I think this should  
> be dealt with as a special case and a less efficient procedure use
> 
> I can't agree with either of your proposals due to performance issues:
> 
> 1) The performance impact of (position, value) is extreme.  I turn  
> all of the BTree O(log N) operations into O(N).  I now have to read  
> all values out of the BTree until I identify the insertion point,  
> then renumber all the following elements.  This defeats the whole  
> purpose of having a BTree in the first place.
> 
> 2) I hate to say it but the CL-SQL approach to cursor operations is  
> not just terribly inefficient, it also defeats the purpose of having  
> indicies on disk by requiring that all values be pulled into memory  
> to support cursor or map operations.  I see the CL-SQL backend as a  
> solution to licensing problems for small databases.  For complex  
> queries over large databases one is better off with an object  
> relational solution like that implemented by CL-SQL's native metaclass.
> 
> The reason that lisp needs to implement a more complex sorting  
> function to support cursor operations over BTrees is that BTrees  
> benefit from ordering types relative to each other (we can't  
> intersperse numeric and non-numeric types without violating lisp  
> ordering) and the cursor fn's need to mirror this.  This doesn't  
> violate lisp ordering, it is additive.  The clarity of the required  
> constraints could be improved, though.
> 
>  > There is a hidden danger in relying upon an order based on the  
> serialized value---namely,
>  > you can now not swap out serializers without drastic side  
> effects.  Since one of the main
>  > ways that we can improve performance is by writing better  
> serializers (and, in particular,
>  > serializers that are specific to particular data types), this  
> seems like a bad idea.
> 
> If designers adhere to the two restrictions above, then a new  
> serializer that supports lisp ordering and introduces a new ordering  
> of types or within types that are unordered in lisp will have no  
> impact on application code.
> 
> Sorry if any point is unclear, I'm in a hurry.  Please ask clarifying  
> questions if necessary.
> 
> Ian
> 
> PS - This leaves me with a question.  Is it possible in any  
> relational DB to register a sorting fn that you can sort by when  
> doing a sorted query?  Typically rows are sorted by their unique ID  
> field (i.e. the key used in the underlying BTree).
> 
> 
> 
> On Aug 2, 2007, at 5:55 PM, Robert L. Read wrote:
> 
> > Yes, I think I understand this.  However, a costly alternative does  
> > exist:  just never let
> > BDB use its own order.  Always impose one that we can compute in  
> > lisp.  Then in
> > BDB you store a (position,value) pair instead of a value, and  
> > either ensure that BDB
> > sorts on the first part of the binary representation of the  
> > position the way you want it to,
> > or you add a lot of logic into the "cursor-next" operation.  This  
> > is how it is done on the
> > CLSQL side.
> >
> > It is almost certainly a bit slower, and it is certainly a bit  
> > harder to code.
> >
> > It seems to me that the root of the problem is that BDB does indeed  
> > order based on a
> > serialized value.  That is what we should remove.  Certainly, if  
> > someone were to write
> > a Pure-LISP backend, which I hope will occur eventually, it would  
> > seem silly for them to
> > have to respect an artifact inherited from BDB, when part of the  
> > purpose of such a project
> > is to escape dependence on BDB.
> >
> > Forgive me if I'm confused but I assert that we should reverse your  
> > argument: we should
> > force BDB to be isomorphic to a lisp sorter, not build a lisp  
> > sorted isomorphic to BDB.
> >
> >
> > On Tue, 2007-07-31 at 15:24 -0400, Ian Eslick wrote:
> >> The practical problem that led to the current design of index  
> >> sorting is that we cannot use lisp code to define the sorting  
> >> function for serialized values inside BDB Btrees (same problem I  
> >> imagine that Henrik had with postmodern). Instead, there is a  
> >> hairy custom C procedure that is registered with BDB that parses  
> >> the serialized format so that sorting is done first by type  
> >> (symbol, string, object, pobject, etc) followed by ordering within  
> >> numeric types, strings and symbols. Everything else is ordered  
> >> based on the byte ordering of its serialized representation. To  
> >> map across indices correctly, we need to know up front whether the  
> >> start value is less than the end value. And so we need a standard  
> >> lisp function that is isomorphic to the BDB sorting function.  
> >> Ideally postmodern would have a similar sorting function that    
> >> properly interprets the serialized format just like the BDB  
> >> function does. I think it's best to have a single standard  
> >> ordering that is as close to lisp's notion of ordering as possible  
> >> so we don't have to maintain different orderings. Ian PS - It  
> >> might be possible to have a lisp ordering function implement BDB's  
> >> notion of sorting by registering it as a callback, however it  
> >> would have to deserialize the BDB values each time. So the  
> >> problems with this are both stability concerns for foreign  
> >> callbacks and the performance impact of serialization/ 
> >> deserialization for internal BDB operations. On the cleanliness/ 
> >> performance axis, I think the current approach is the right  
> >> tradeoff (it's the original one Ben made, FYI). On Jul 31, 2007,  
> >> at 12:50 PM, Robert L. Read wrote: > Personally, I think the only  
> >> sensible way to handle this problem is > to require the user to >  
> >> specify an ordering function. We can of course provide a default,  
> >> > which will be error-prone > but tend to work most of the time. >  
> >> > The function called "my-generic-less-than" which is in the  
> >> source > tree now could be > a starting point for a generic  
> >> ordering. > > > On Tue, 2007-07-24 at 09:48 -0400, Ian Eslick  
> >> wrote: >> Robert and I have had some extended discussions on  
> >> ordering in >> indices. I think that all we really need to agree  
> >> on is _some_ >> canonical ordering. If we have mixed types in an  
> >> index, how should >> they be ordered relative to each other? In  
> >> BDB we have a C >> function which implements the ordering based on  
> >> the type tag and >> then based on the type within it. Are you  
> >> relying on a pure binary >> sort in postmodern? Robert or I will  
> >> get to submitting that patch >> shortly. I have recently sent in a  
> >> patch to lisp-compare<= so >> we'll see if we had to make parallel  
> >> changes. Thanks, Ian On Jul >> 24, 2007, at 3:50 AM, Henrik Hjelte  
> >> wrote: > I sent this message >> yesterday but I guess it got stuck  
> >> in the mailing > list filter. >> Perhaps the attachment was too  
> >> big. Since my > common-lisp.net >> user hhjelte does not have  
> >> write access to elephant I > have >> placed the patches from here  
> >> instead: > darcs get http://common- >> lisp.net/project/grand-prix/ 
> >> darcs/elephant > > ---------- >> Forwarded message ---------- >  
> >> From: Henrik Hjelte >> <henrik at evahjelte.com> > Date: Jul 23, 2007  
> >> 11:28 PM > Subject: >> some patches > To: elephant-devel at common- 
> >> lisp.net > > > Here are >> some darcs patches that might be of  
> >> interest. I had some > >> problems with map-index on db-postmodern  
> >> that made me almost rip >> my > hair of, but finally I made it to  
> >> work again. The problem is >> that > map-index for a string value  
> >> rely on the ordering in the >> btree > (continue-p makes use of  
> >> less than for strings). The >> postmodern > backend relies on how  
> >> the database backend orders >> things, which is not > always the  
> >> same thing. Is it a necessary >> feature that b-trees of > string  
> >> and objects are required to be >> ordered by lisp-compare<=? > >  
> >> In the process of solving the bug I >> have upgraded the test  
> >> framework > to use FiveAM instead of RT, It >> has in my opinion a  
> >> very nice syntax > and some useful features to >> track  
> >> dependencies between tests. I hope > you agree that it >> improves  
> >> on things. > > /Henrik Hjelte > >>  
> >> _______________________________________________ > elephant-devel  
> >> >> site list > elephant-devel at common-lisp.net > http://common- >>  
> >> lisp.net/mailman/listinfo/elephant-devel >>  
> >> _______________________________________________ elephant-devel >>  
> >> site list elephant-devel at common-lisp.net http://common-lisp.net/  
> >> >> mailman/listinfo/elephant-devel >  
> >> _______________________________________________ > elephant-devel  
> >> site list > elephant-devel at common-lisp.net > http://common- 
> >> lisp.net/mailman/listinfo/elephant-devel  
> >> _______________________________________________ elephant-devel  
> >> site list elephant-devel at common-lisp.net http://common-lisp.net/ 
> >> mailman/listinfo/elephant-devel
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070803/643eb563/attachment.html>

From read at robertlread.net  Thu Aug  2 23:21:23 2007
From: read at robertlread.net (read at robertlread.net)
Date: Thu, 2 Aug 2007 18:21:23 -0500
Subject: [elephant-devel] sqlite backend, "Error 5 / database is locke
Message-ID: <200708022321.l72NLNZu028561@franklin.robertlread.net>


Forgive me taking a long time to answer this; I have been on traveling.

Here is the comment that I wrote into the sql-controller.lisp file, more than a year ago, I guess:

;; Every actual CL-SQL connection has to be in a separate thread.
;; My solution to this is to keep a map of threads, and reuse connections within a certain thread.

That is, CL-SQL itself is threadsafe if you use a separate connection in each thread.  You have to 
provide a mutex so that multiple threads don't access the same connection simultaneously.
As far as I know, this all functions correctly in Elephant and the user doesn't have to worry about 
it at all; certainly my own application is multi-threaded and I have not noticed a problem, although
that is not a very good stress test.

However, I think I am agreeing with the author in saying that the problem could well be inside
SQLite3 itself.

My memory of all of this stuff is a little rusty; I personally am not sure how robust SQLite is in
a multi-threaded environment.




From henrik at evahjelte.com  Thu Aug  9 20:32:14 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Thu, 9 Aug 2007 22:32:14 +0200
Subject: [elephant-devel] postmodern update
Message-ID: <50e8e4f60708091332g782472dfh1b7c3f0311c2c13d@mail.gmail.com>

Some problem solved with the postmodern backend, there are patches in this repo:

darcs get http://common-lisp.net/project/grand-prix/darcs/elephant

/Henrik


From henrik at evahjelte.com  Tue Aug 14 11:54:12 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Tue, 14 Aug 2007 13:54:12 +0200
Subject: [elephant-devel] Re: postmodern update
In-Reply-To: <50e8e4f60708091332g782472dfh1b7c3f0311c2c13d@mail.gmail.com>
References: <50e8e4f60708091332g782472dfh1b7c3f0311c2c13d@mail.gmail.com>
Message-ID: <50e8e4f60708140454g314a4f7ci874da7fa90728607@mail.gmail.com>

On 8/9/07, Henrik Hjelte <henrik at evahjelte.com> wrote:
> Some problem solved with the postmodern backend, there are patches in this repo:
>
> darcs get http://common-lisp.net/project/grand-prix/darcs/elephant
>
> /Henrik
>

I have added some more patches.
/Henrik


From read at robertlread.net  Tue Aug 14 14:10:19 2007
From: read at robertlread.net (Robert L. Read)
Date: Tue, 14 Aug 2007 09:10:19 -0500
Subject: [elephant-devel] Re: postmodern update
In-Reply-To: <50e8e4f60708140454g314a4f7ci874da7fa90728607@mail.gmail.com>
References: <50e8e4f60708091332g782472dfh1b7c3f0311c2c13d@mail.gmail.com>
	<50e8e4f60708140454g314a4f7ci874da7fa90728607@mail.gmail.com>
Message-ID: <1187100619.5895.134.camel@penguin.yourdomain.com>

Thanks.

I hope to get these integrated and tested by the end of the week.



On Tue, 2007-08-14 at 13:54 +0200, Henrik Hjelte wrote:
> On 8/9/07, Henrik Hjelte <henrik at evahjelte.com> wrote:
> > Some problem solved with the postmodern backend, there are patches in this repo:
> >
> > darcs get http://common-lisp.net/project/grand-prix/darcs/elephant
> >
> > /Henrik
> >
> 
> I have added some more patches.
> /Henrik
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From jcorneli at planetmath.org  Tue Aug 14 22:14:30 2007
From: jcorneli at planetmath.org (Joe Corneli)
Date: Tue, 14 Aug 2007 18:14:30 -0400 (EDT)
Subject: [elephant-devel] a job for the query system?
Message-ID: <20070814221430.AB4F0826D@planetmath.cc.vt.edu>


I want to work with some objects like this:

 Things --
 [key: OID | slots: name, data]

 Triples (subclass of Thing) --
 [slots: beginning, middle, end]

 Theories (subclass of Thing) --
 pset (?) of elements: [key: name | slot: OID of some Thing]

Ian Eslick has already helped me by putting together an implementation
covering Things and Triples in a very slick way.

 http://planetmath.org/~jcorneli/variant-4.lisp

In particular, the current implementation enables the user to find all
Triples which match on any particular data in the three slots.  Note:
the schema is not precisely what I've listed above, which brings me to
my first question:

Question 1.  What should I do to key a persistent class by its OID?

Theories have yet to be implemented.  The schema is set up so that
Things to live in a Universe and Theories designate arbitrary subsets
of that Universe.  The challenge that I want the data-matching
routines to work within a given Theory.  This way, a Theory can be
viewed as a sub-network of the semantic net that the Universe as a
whole comprises.

I am pretty sure Ian indicated that this can be done using the up and
coming "query system".  Which brings me to:

Question 2.  Does this indeed look like something the query system can
handle?

Question 3.  Does this look like a good set-up?

Question 3.  How can I learn more about this query system and when is
it expected to be available?




From read at robertlread.net  Fri Aug 17 19:46:46 2007
From: read at robertlread.net (Robert L. Read)
Date: Fri, 17 Aug 2007 14:46:46 -0500
Subject: [elephant-devel] Testing Henrik's latest patches....
Message-ID: <1187380006.15073.247.camel@penguin.yourdomain.com>


I've tested Henrik's latest patches to the postmodern back-end.
They are green on my 64-bit linux system.

Unfortunately, I am not sure that this version is compatible with the
previous (not officially released version.)

Perhaps foolishly, I went ahead and deployed the previous version to 
my live site, even though we have not officially released it, so I 
now have to delve into what the issues may be.

Probably by the end of next week I will have committed his patches to
the elephant repository.





From eslick at csail.mit.edu  Fri Aug 31 01:45:34 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Thu, 30 Aug 2007 21:45:34 -0400
Subject: [elephant-devel] a job for the query system?
In-Reply-To: <20070814221430.AB4F0826D@planetmath.cc.vt.edu>
References: <20070814221430.AB4F0826D@planetmath.cc.vt.edu>
Message-ID: <8D1E5530-E451-42EB-A216-6E835C16D347@csail.mit.edu>


On Aug 14, 2007, at 6:14 PM, Joe Corneli wrote:

>
> I want to work with some objects like this:
>
>  Things --
>  [key: OID | slots: name, data]
>
>  Triples (subclass of Thing) --
>  [slots: beginning, middle, end]
>
>  Theories (subclass of Thing) --
>  pset (?) of elements: [key: name | slot: OID of some Thing]
>
> Ian Eslick has already helped me by putting together an implementation
> covering Things and Triples in a very slick way.
>
>  http://planetmath.org/~jcorneli/variant-4.lisp
>
> In particular, the current implementation enables the user to find all
> Triples which match on any particular data in the three slots.  Note:
> the schema is not precisely what I've listed above, which brings me to
> my first question:
>
> Question 1.  What should I do to key a persistent class by its OID?

These questions may be too low level and making too many assumptions.

A persistent class can be indexed in many different ways.  By  
default, an indexed class (has class option :index t or any slot has  
option :index t) is stored in an index by its OID.  Currently we do  
not guarantee that OIDs will not change in a migration or GC and  
there is some open discussion about providing a long-lived namespace  
(probably when we move to 64-bit OIDs).

You can create your own OIDs, put persistent objects into BTrees  
directly and ignore all the class indexing stuff.  If you want to use  
the system assigned OID, you can access it directly by (elephant::oid  
<thing obj>)

However, unless you are building a specialized data structure like a  
map, you can just use the object itself as a key in lisp (internally  
this is stored as the object's OID, but it makes things cleaner in  
lisp and ensures consistency across migrations, GCs, etc).  So your  
schema would be:

(class:Thing slot:name slot:data)
(class:Triple slot:beginning slot:middle slot:end)
(class:Theory slot:name slot:<pset ref>)

This pset would have a set of <thing ref>, or instances of things  
which I assume would be triples.

Thus when you iterate over a pset, you get back <thing ref>'s instead  
of OIDs.  This does have some implications, namely creating the thing  
object in memory rather than deserializing an integer, but the query  
system in theory should overcome this.

> Theories have yet to be implemented.  The schema is set up so that
> Things to live in a Universe and Theories designate arbitrary subsets
> of that Universe.  The challenge that I want the data-matching
> routines to work within a given Theory.  This way, a Theory can be
> viewed as a sub-network of the semantic net that the Universe as a
> whole comprises.

> I am pretty sure Ian indicated that this can be done using the up and
> coming "query system".  Which brings me to:
>
> Question 2.  Does this indeed look like something the query system can
> handle?

If I understand you, you want to filter the lookup of a triple or  
thing to be those matching the query criteria but also contained  
within a specific theory?  Expressed in Psuedo-SQL

SELECT triple WHERE
triple.beginning = <Thing name:IsA> AND
triple.end = <Thing name:Cat> AND
member(triple, <Theory name:QuantumTheory>)

This would require that the query system be able to do membership  
tests on psets, if you used the proposed schema then I think this  
wouldn't require a significant change to the query system (just  
another join on the pset index).  The pset functions as a one-to-many  
map, but it can also be seen from the SQL perspsective as a virtual  
column that can be added to any table that provides boolean membership.

> Question 3.  Does this look like a good set-up?

See my points above.

> Question 3.  How can I learn more about this query system and when is
> it expected to be available?

There is some early prototype functionality that is partially in my  
local development tree and partially in the current development  
tree.  Unfortunately I did not finish the query system this summer  
and so its release has been indefinitely delayed.  I will try to get  
the sketch with a basic interface released in the next month, but  
optimizations may be long in coming.

Good luck!
Ian