From MR at common-lisp.net  Fri Jan  5 17:15:42 2007
From: MR at common-lisp.net (MR at common-lisp.net)
Date: Fri, 05 Jan 2007 18:15:42 +0100
Subject: [elephant-devel] [SPAM] PROPOSAL FROM MR,
	BABABELLO AKU(URGENT RESPOND AND CONFIDENTAL
Message-ID: <20070105171543.BC4D01E072@common-lisp.net>

An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070105/4b85e927/attachment.ksh>
-------------- next part --------------
An embedded message was scrubbed...
From: MR at common-lisp.net, BABABELLO AKU <mr_aku206 at yahoo.es>
Subject: PROPOSAL FROM MR, BABABELLO AKU(URGENT RESPOND AND CONFIDENTAL
Date: Fri, 05 Jan 2007 18:15:42 +0100
Size: 2878
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070105/4b85e927/attachment.mht>

From cjstuij at gmail.com  Sun Jan 14 10:43:39 2007
From: cjstuij at gmail.com (Ties Stuij)
Date: Sun, 14 Jan 2007 11:43:39 +0100
Subject: [elephant-devel] WARNING
In-Reply-To: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
Message-ID: <d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>

I wanted to test the serializer changes so i got elephant from cvs,
but elephant.asd complains that there are files missing:
cross-platform.lisp, serializer1.lisp, serializer2.lisp and
unicode2.lisp.

Greets,
Ties


On 12/16/06, Ian Eslick <eslick at csail.mit.edu> wrote:
> I just made a large checkin to HEAD.  This means that HEAD is now
> BROKEN!!
>
> This checkpoint is part of a large change including abstracting the
> serializer backend, supporting 64-bit lisps, a new method of
> versioning and a host of other 0.6.1 related changes.
>
> I hope to correct or document the current problems later today so
> other developers can review or improve the features for 0.6.1.
>
> Thank you,
> Ian
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>


From read at robertlread.net  Sun Jan 14 16:47:46 2007
From: read at robertlread.net (Robert L. Read)
Date: Sun, 14 Jan 2007 10:47:46 -0600
Subject: [elephant-devel] WARNING
In-Reply-To: <d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
	<d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
Message-ID: <1168793266.5315.37.camel@localhost.localdomain>

Ian Eslick is in the middle of a major rewrite; I'm afraid the head may
not be
in a consistent state at the moment.

I assume Ian will answer this when he gets to it.

His rewrite was interrupted by a significant and unexpected, but joyous,
event.

On Sun, 2007-01-14 at 11:43 +0100, Ties Stuij wrote:

> I wanted to test the serializer changes so i got elephant from cvs,
> but elephant.asd complains that there are files missing:
> cross-platform.lisp, serializer1.lisp, serializer2.lisp and
> unicode2.lisp.
> 
> Greets,
> Ties
> 
> 
> On 12/16/06, Ian Eslick <eslick at csail.mit.edu> wrote:
> > I just made a large checkin to HEAD.  This means that HEAD is now
> > BROKEN!!
> >
> > This checkpoint is part of a large change including abstracting the
> > serializer backend, supporting 64-bit lisps, a new method of
> > versioning and a host of other 0.6.1 related changes.
> >
> > I hope to correct or document the current problems later today so
> > other developers can review or improve the features for 0.6.1.
> >
> > Thank you,
> > Ian
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070114/3dd00ee2/attachment.html>

From eslick at csail.mit.edu  Sun Jan 14 17:09:54 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sun, 14 Jan 2007 12:09:54 -0500
Subject: [elephant-devel] WARNING
In-Reply-To: <d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
	<d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
Message-ID: <1E6058D1-06DF-4C67-BEE0-94EFEC7DF434@csail.mit.edu>

Sorry, I should update the main page.  CVS is currently broken due to  
a delayed in-progress change.  It should be fixed in the next few  
weeks.  CVS as of 12/1/2006 should work fine or stick with the formal  
0.6.0 release unless you really need one of the new features.

I'm actually back and working on the promised Elephant checkin now.

Ian

On Jan 14, 2007, at 5:43 AM, Ties Stuij wrote:

> I wanted to test the serializer changes so i got elephant from cvs,
> but elephant.asd complains that there are files missing:
> cross-platform.lisp, serializer1.lisp, serializer2.lisp and
> unicode2.lisp.
>
> Greets,
> Ties
>
>
> On 12/16/06, Ian Eslick <eslick at csail.mit.edu> wrote:
>> I just made a large checkin to HEAD.  This means that HEAD is now
>> BROKEN!!
>>
>> This checkpoint is part of a large change including abstracting the
>> serializer backend, supporting 64-bit lisps, a new method of
>> versioning and a host of other 0.6.1 related changes.
>>
>> I hope to correct or document the current problems later today so
>> other developers can review or improve the features for 0.6.1.
>>
>> Thank you,
>> Ian
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From cjstuij at gmail.com  Sun Jan 14 19:03:55 2007
From: cjstuij at gmail.com (Ties Stuij)
Date: Sun, 14 Jan 2007 20:03:55 +0100
Subject: [elephant-devel] WARNING
In-Reply-To: <1E6058D1-06DF-4C67-BEE0-94EFEC7DF434@csail.mit.edu>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
	<d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
	<1E6058D1-06DF-4C67-BEE0-94EFEC7DF434@csail.mit.edu>
Message-ID: <d7ce4ebf0701141103ke0478a6r2a46de09096760c5@mail.gmail.com>

On 1/14/07, Ian Eslick <eslick at csail.mit.edu> wrote:
> Sorry, I should update the main page.  CVS is currently broken due to
> a delayed in-progress change.  It should be fixed in the next few
> weeks.  CVS as of 12/1/2006 should work fine or stick with the formal
> 0.6.0 release unless you really need one of the new features.
>
> I'm actually back and working on the promised Elephant checkin now.
>
> Ian

ah, thanks for the reply. And congratulations with your joyous event,
Ties


From henrik at evahjelte.com  Fri Jan 19 19:24:38 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Fri, 19 Jan 2007 20:24:38 +0100
Subject: [elephant-devel] WARNING
In-Reply-To: <1168793266.5315.37.camel@localhost.localdomain>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
	<d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
	<1168793266.5315.37.camel@localhost.localdomain>
Message-ID: <1169234678.7847.45.camel@trinidad>

I saw that some new files were checked in, so I
tried to get the new version going. This is how far I've got so far:

One file missing:
src/db-bdb/bdb-symbol-tables.lisp

Some defgeneric forms in controller had ((sc store-controller)) as
argument instead of (store-controller).

When compiling against the Berkey DB 4.4.20 (the latest generation 4.4 i
found), I had to comment out the function db_compact in libberkeley-db.c
in order to make it compile.

I have a small suggestion to ele-bdb.asd below:

(defun get-config-option (option component)
  (unless *bdb-config*
    (let ((filespec (make-pathname :defaults (asdf:component-pathname  

(asdf:component-system component))
                                           :name "my-config"
                                           :type "sexp")))
      (unless (probe-file filespec)
        (error "Missing file. Copy config.sexp in elephant root
directory to my-config.sexp and edit it appropriately."))
      (with-open-file (config filespec)
        (setf *bdb-config* (read config)))))
  (cdr (assoc option *bdb-config*)))


I have also found a bug in clsql-collections, the method
cursor-pset. Position should have a :test #'equal to make it work with
strings.

(defmethod cursor-pset ((cursor sql-secondary-cursor) key)
  (declare (optimize (speed 3)))
  (unless (cursor-initialized-p cursor)
    (cursor-init cursor))
  (let ((idx (position key (:sql-crsr-ks cursor) :test #'equal)))
    (if idx
        (progn
          (setf (:sql-crsr-ck cursor) idx)
          (setf (:dp-nmbr cursor) 0)
          (cursor-current-x cursor :returnpk t))
        (cursor-un-init cursor)
        )))

And congratulations on the joyful event. Was it Christmas?

/Henrik Hjelte





From eslick at csail.mit.edu  Fri Jan 19 20:17:15 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Fri, 19 Jan 2007 15:17:15 -0500
Subject: [elephant-devel] WARNING
In-Reply-To: <1169234678.7847.45.camel@trinidad>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
	<d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
	<1168793266.5315.37.camel@localhost.localdomain>
	<1169234678.7847.45.camel@trinidad>
Message-ID: <2EA43711-AE00-4661-B7A8-3E922D9CA6FB@csail.mit.edu>

Henrik,

I'm very sorry for the state of CVS and the delay in getting it put  
back together.  Once I have my local copy functioning again, I'll  
download the latest HEAD and make sure it works, at least, with Mac  
and Allegro.  I promise to send a mail to the list when HEAD is  
(mostly) working again.

As for the aforementioned event: my wife gave birth to twin daughters  
about a week ago and was on bed rest a few weeks before that; my  
hacking time was more severely curtailed than I'd hoped.  Of course  
after the birth we discovered new extremes of sleep deprivation and  
I'm sure no one wants to see the code I would have generated then!

I'll roll your suggestions and bug fixes into my local copy.

Regards,
Ian


On Jan 19, 2007, at 2:24 PM, Henrik Hjelte wrote:

> I saw that some new files were checked in, so I
> tried to get the new version going. This is how far I've got so far:
>
> One file missing:
> src/db-bdb/bdb-symbol-tables.lisp
>
> Some defgeneric forms in controller had ((sc store-controller)) as
> argument instead of (store-controller).
>
> When compiling against the Berkey DB 4.4.20 (the latest generation  
> 4.4 i
> found), I had to comment out the function db_compact in libberkeley- 
> db.c
> in order to make it compile.
>
> I have a small suggestion to ele-bdb.asd below:
>
> (defun get-config-option (option component)
>   (unless *bdb-config*
>     (let ((filespec (make-pathname :defaults (asdf:component-pathname
>
> (asdf:component-system component))
>                                            :name "my-config"
>                                            :type "sexp")))
>       (unless (probe-file filespec)
>         (error "Missing file. Copy config.sexp in elephant root
> directory to my-config.sexp and edit it appropriately."))
>       (with-open-file (config filespec)
>         (setf *bdb-config* (read config)))))
>   (cdr (assoc option *bdb-config*)))
>
>
> I have also found a bug in clsql-collections, the method
> cursor-pset. Position should have a :test #'equal to make it work with
> strings.
>
> (defmethod cursor-pset ((cursor sql-secondary-cursor) key)
>   (declare (optimize (speed 3)))
>   (unless (cursor-initialized-p cursor)
>     (cursor-init cursor))
>   (let ((idx (position key (:sql-crsr-ks cursor) :test #'equal)))
>     (if idx
>         (progn
>           (setf (:sql-crsr-ck cursor) idx)
>           (setf (:dp-nmbr cursor) 0)
>           (cursor-current-x cursor :returnpk t))
>         (cursor-un-init cursor)
>         )))
>
> And congratulations on the joyful event. Was it Christmas?
>
> /Henrik Hjelte
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Fri Jan 19 20:19:31 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Fri, 19 Jan 2007 15:19:31 -0500
Subject: [elephant-devel] WARNING
In-Reply-To: <2EA43711-AE00-4661-B7A8-3E922D9CA6FB@csail.mit.edu>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
	<d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
	<1168793266.5315.37.camel@localhost.localdomain>
	<1169234678.7847.45.camel@trinidad>
	<2EA43711-AE00-4661-B7A8-3E922D9CA6FB@csail.mit.edu>
Message-ID: <83BA8C31-B238-4643-AC3C-1F8EB739DE5B@csail.mit.edu>

I should have said she gave birth six weeks ago...

On Jan 19, 2007, at 3:17 PM, Ian Eslick wrote:

> Henrik,
>
> I'm very sorry for the state of CVS and the delay in getting it put  
> back together.  Once I have my local copy functioning again, I'll  
> download the latest HEAD and make sure it works, at least, with Mac  
> and Allegro.  I promise to send a mail to the list when HEAD is  
> (mostly) working again.
>
> As for the aforementioned event: my wife gave birth to twin  
> daughters about a week ago and was on bed rest a few weeks before  
> that; my hacking time was more severely curtailed than I'd hoped.   
> Of course after the birth we discovered new extremes of sleep  
> deprivation and I'm sure no one wants to see the code I would have  
> generated then!
>
> I'll roll your suggestions and bug fixes into my local copy.
>
> Regards,
> Ian
>
>
> On Jan 19, 2007, at 2:24 PM, Henrik Hjelte wrote:
>
>> I saw that some new files were checked in, so I
>> tried to get the new version going. This is how far I've got so far:
>>
>> One file missing:
>> src/db-bdb/bdb-symbol-tables.lisp
>>
>> Some defgeneric forms in controller had ((sc store-controller)) as
>> argument instead of (store-controller).
>>
>> When compiling against the Berkey DB 4.4.20 (the latest generation  
>> 4.4 i
>> found), I had to comment out the function db_compact in  
>> libberkeley-db.c
>> in order to make it compile.
>>
>> I have a small suggestion to ele-bdb.asd below:
>>
>> (defun get-config-option (option component)
>>   (unless *bdb-config*
>>     (let ((filespec (make-pathname :defaults (asdf:component-pathname
>>
>> (asdf:component-system component))
>>                                            :name "my-config"
>>                                            :type "sexp")))
>>       (unless (probe-file filespec)
>>         (error "Missing file. Copy config.sexp in elephant root
>> directory to my-config.sexp and edit it appropriately."))
>>       (with-open-file (config filespec)
>>         (setf *bdb-config* (read config)))))
>>   (cdr (assoc option *bdb-config*)))
>>
>>
>> I have also found a bug in clsql-collections, the method
>> cursor-pset. Position should have a :test #'equal to make it work  
>> with
>> strings.
>>
>> (defmethod cursor-pset ((cursor sql-secondary-cursor) key)
>>   (declare (optimize (speed 3)))
>>   (unless (cursor-initialized-p cursor)
>>     (cursor-init cursor))
>>   (let ((idx (position key (:sql-crsr-ks cursor) :test #'equal)))
>>     (if idx
>>         (progn
>>           (setf (:sql-crsr-ck cursor) idx)
>>           (setf (:dp-nmbr cursor) 0)
>>           (cursor-current-x cursor :returnpk t))
>>         (cursor-un-init cursor)
>>         )))
>>
>> And congratulations on the joyful event. Was it Christmas?
>>
>> /Henrik Hjelte
>>
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Fri Jan 19 20:20:05 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Fri, 19 Jan 2007 15:20:05 -0500
Subject: [elephant-devel] WARNING
In-Reply-To: <1169234678.7847.45.camel@trinidad>
References: <6B94CB17-A45F-4880-B475-E173CF252D7F@csail.mit.edu>
	<d7ce4ebf0701140243q6c67cb09we7201b28a84f27ac@mail.gmail.com>
	<1168793266.5315.37.camel@localhost.localdomain>
	<1169234678.7847.45.camel@trinidad>
Message-ID: <6C283BDD-A720-4672-AFBE-D3F7921DF6D8@csail.mit.edu>

Henrik,

I'm very sorry for the state of CVS and the delay in getting it put  
back together.  Once I have my local copy functioning again, I'll  
download the latest HEAD and make sure it works, at least, with Mac  
and Allegro.  I promise to send a mail to the list when HEAD is  
(mostly) working again.

As for the aforementioned event: my wife gave birth to twin daughters  
about a week ago and was on bed rest a few weeks before that; my  
hacking time was more severely curtailed than I'd hoped.  Of course  
after the birth we discovered new extremes of sleep deprivation and  
I'm sure no one wants to see the code I would have generated then!

I'll roll your suggestions and bug fixes into my local copy.

Regards,
Ian


On Jan 19, 2007, at 2:24 PM, Henrik Hjelte wrote:

> I saw that some new files were checked in, so I
> tried to get the new version going. This is how far I've got so far:
>
> One file missing:
> src/db-bdb/bdb-symbol-tables.lisp
>
> Some defgeneric forms in controller had ((sc store-controller)) as
> argument instead of (store-controller).
>
> When compiling against the Berkey DB 4.4.20 (the latest generation  
> 4.4 i
> found), I had to comment out the function db_compact in libberkeley- 
> db.c
> in order to make it compile.
>
> I have a small suggestion to ele-bdb.asd below:
>
> (defun get-config-option (option component)
>   (unless *bdb-config*
>     (let ((filespec (make-pathname :defaults (asdf:component-pathname
>
> (asdf:component-system component))
>                                            :name "my-config"
>                                            :type "sexp")))
>       (unless (probe-file filespec)
>         (error "Missing file. Copy config.sexp in elephant root
> directory to my-config.sexp and edit it appropriately."))
>       (with-open-file (config filespec)
>         (setf *bdb-config* (read config)))))
>   (cdr (assoc option *bdb-config*)))
>
>
> I have also found a bug in clsql-collections, the method
> cursor-pset. Position should have a :test #'equal to make it work with
> strings.
>
> (defmethod cursor-pset ((cursor sql-secondary-cursor) key)
>   (declare (optimize (speed 3)))
>   (unless (cursor-initialized-p cursor)
>     (cursor-init cursor))
>   (let ((idx (position key (:sql-crsr-ks cursor) :test #'equal)))
>     (if idx
>         (progn
>           (setf (:sql-crsr-ck cursor) idx)
>           (setf (:dp-nmbr cursor) 0)
>           (cursor-current-x cursor :returnpk t))
>         (cursor-un-init cursor)
>         )))
>
> And congratulations on the joyful event. Was it Christmas?
>
> /Henrik Hjelte
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From mega at retes.hu  Sat Jan 20 18:55:50 2007
From: mega at retes.hu (=?iso-8859-1?q?G=E1bor_Melis?=)
Date: Sat, 20 Jan 2007 19:55:50 +0100
Subject: [elephant-devel] defaults
Message-ID: <200701201955.51390.mega@retes.hu>

Hello

The attached patch (against 0.6.0) does two things:

* changes the default locations of the db4.3 on linux to what I guess 
should match what most users have and the filesystem hierarchy standard

* guards against loading the pthreads library on threaded sbcl where the 
right version is already loaded and it crashes sbcl

Cheers, Gabor Melis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: default.diff
Type: text/x-diff
Size: 1410 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070120/5be6efa3/attachment.diff>

From mega at retes.hu  Sat Jan 20 18:57:52 2007
From: mega at retes.hu (=?iso-8859-1?q?G=E1bor_Melis?=)
Date: Sat, 20 Jan 2007 19:57:52 +0100
Subject: [elephant-devel] checkpoint
Message-ID: <200701201957.52833.mega@retes.hu>

The attached patch adds support DB_ENV.txn_checkpoint that's useful if 
one wants to schedule checkpointing from lisp.

Regards, Gabor Melis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: checkpoint.diff
Type: text/x-diff
Size: 1701 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070120/432ac78e/attachment.diff>

From mega at retes.hu  Sat Jan 20 19:42:22 2007
From: mega at retes.hu (=?iso-8859-1?q?G=E1bor_Melis?=)
Date: Sat, 20 Jan 2007 20:42:22 +0100
Subject: [elephant-devel] run-elephant-thread
Message-ID: <200701202042.22816.mega@retes.hu>

The fine manual at 
http://common-lisp.net/project/elephant/doc/Threading.html says that 
run-elephant-thread is broken but leaves the consequences to my 
imagination.

Considering the comment about specials for buffers and such as well, my 
reading is that there is no official way of using elephant from 
multithreaded code.

Is that right?
What needs to be done to make run-elephant-thread safe again?

Gabor Melis


From read at robertlread.net  Sat Jan 20 20:06:40 2007
From: read at robertlread.net (Robert L. Read)
Date: Sat, 20 Jan 2007 14:06:40 -0600
Subject: [elephant-devel] run-elephant-thread
In-Reply-To: <200701202042.22816.mega@retes.hu>
References: <200701202042.22816.mega@retes.hu>
Message-ID: <1169323600.4833.11.camel@localhost.localdomain>

Ian is working on this; I think significant progress will be made when
he finishes his check-in.

The biggest problem is that serializer uses globals in such a way that
it is not thread-safe;
Ian is specifically rewriting that.

I personally solve problem by using SBCL mutexes around the
serialize/deserialize code, 
but I have not committed that code, since it is highly SBCL-specific,
and in fact I don't 
know of a good way to do locking portably.

The CL-SQL backend also has a problem with multi-threading.  I have a
local, SBCL solution
that uses a hash-map to create a separate connection for each thread in
which a controller
is used.  I have not committed this for the same reasons (and a touch of
laziness.)

So to answer your question fully:

1)  For BDB-based usage, we need to wait for Ian to finish the work that
currently is underway.
2)  For CL-SQL based usage, we need some sort of portable thread/lock
solution, or at least
a switch that says "for SBCL, do this", and we need for me, or somebody
else, to supply that 
code.

Can you recommend a portable approach to the threading problem?   I
personally only use SBCL,
and I think Ian uses Allegro.  We can stand on our heads if we have to
to keep SBCL both 
portable and threadsafe, but I need expert advice on dealing with this.

And, BTW, I doubt the serializer was ever threadsafe.  If you use BDB,
you can go along time
before you ever notice a problem; but of course we want an enterprise-
quality system and so 
must have complete thead-safety eventually.




On Sat, 2007-01-20 at 20:42 +0100, G?bor Melis wrote:

> The fine manual at 
> http://common-lisp.net/project/elephant/doc/Threading.html says that 
> run-elephant-thread is broken but leaves the consequences to my 
> imagination.
> 
> Considering the comment about specials for buffers and such as well, my 
> reading is that there is no official way of using elephant from 
> multithreaded code.
> 
> Is that right?
> What needs to be done to make run-elephant-thread safe again?
> 
> Gabor Melis
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070120/adb97d75/attachment.html>

From eslick at csail.mit.edu  Sat Jan 20 21:54:42 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sat, 20 Jan 2007 16:54:42 -0500
Subject: [elephant-devel] run-elephant-thread
In-Reply-To: <200701202042.22816.mega@retes.hu>
References: <200701202042.22816.mega@retes.hu>
Message-ID: <A9498CEB-77D1-4239-8D8E-875373A4FBDF@csail.mit.edu>

There should be an e-mail about this in the archive somewhere, but to  
summarize...

I think that requiring client code to worry about wrapping every  
thread in run-elephant-thread is unrealistic, so that interface is  
now deprecated.  The reason for this is simple, when you are using a  
multithreaded web server the main thread launches client threads  
inside the server code and there is no way to wrap it in an elephant  
thread without modifying the web server and this is unrealistic.

A Thread-Safe Serializer:

The answer to this problem is a little more important than the  
others.  The serializer is called all the time and is a performance  
critical part of the system.  The serializer is thread safe except  
for it's use of a hash table used to detect circular data  
structures.  I've used elephant in a multi-threaded setting for quite  
some time without worrying about the serializer because in practice I  
would never hit the case where an object I was serializing would  
accidentally lookup a duplicate object/id in the circularity hash.   
The way to avoid this case in general is to have a queue of hash  
tables that each thread can grab when it wants to serialize an object  
(you don't want to allocate a hash table in an inner loop - reuse by  
clearing is also costly, but about 50% as much).  So only this queue  
needs to be protected.

I tried using standard locks in the various EXCL-like extended  
packages but the performance is atrocious for frequently called  
routines like serialize.  Instead, I use without-interrupts (a common  
lisp primitive) to block interrupts for the duration of a vector-pop  
command that grabs a new hash.  This gives much better performance.   
The only other variables that need to be so protected are:

Thread Safety in Backends:

I'm pretty sure that the current behavior of BDB is thread-safe.  I  
researched this earlier, but only remember that I concluded it was  
safe so if anyone remembers the details feel free to contradict me.

A quick Google investigation says that CL-SQL requires, at a minimum,  
that each thread have it's own connection object to be thread safe,  
so each thread needs to reconnect to the cl-sql database plus have a  
thread-local clsql:*default-database* binding.  (I don't think this  
works for SQLite though)

These are pretty easy and will be handled in my next checkin:
-------------------------------------------------------------

Global variables (infrequently written):
*elephant-controller-init*
*dbconnection-spec*

Store-controller slots that need infrequent write-protection:
- instance-cache
- symbol-cache (0.6.1+)


The following elephant variables are a little tricky:
-----------------------------------------------------

Thread-local global vars (frequently accessed):
*store-controller* (if different threads use different controllers)
*transaction-stack*
*current-transaction*
*resourced-byte-spec*
(errno handling in uffi?)

Deprecated thread-local vars:
*auto-commit* (BDB 4.4 no longer pays attention to auto-commit  
arguments so we can remove this from elephant)

1) You can use the macro with-elephant-variables in 0.6.1 to create  
new, thread-local dynamic bindings of the above variables, but that  
is still a manual solution for when you have access to the thread  
creation code and can create thread-local specials.

2) A more consequential is to excise required dependency on these  
variables entirely.  The implication of this is a potentially  
significant API change where an application can always provide the  
store controller on calls to collection accessors, cursor operations,  
etc and it defaults to *store-controller* for environments where  
there is only one store or where the user is managing the binding of  
*store-controller* in each thread.  I think this is already  
accommodated in much of the API, but I haven't investigated this to  
see how much work it is.

We can further require that all transactions be wrapped in 'with- 
elephant-transaction' so that the appropriate specials are  
dynamically bound within the stack.  I think this actually would be  
pretty easy.  We could document the internals of with-elephant- 
transaction for anyone who wants to do something sophisticated and is  
willing to manage the thread issues themselves.

Does anyone have a better suggestion here?  For example is there a  
portability layer that can detect the current thread ID and use that  
to index the default global values?

Regards,
Ian

On Jan 20, 2007, at 2:42 PM, G?bor Melis wrote:

> The fine manual at
> http://common-lisp.net/project/elephant/doc/Threading.html says that
> run-elephant-thread is broken but leaves the consequences to my
> imagination.
>
> Considering the comment about specials for buffers and such as  
> well, my
> reading is that there is no official way of using elephant from
> multithreaded code.
>
> Is that right?
> What needs to be done to make run-elephant-thread safe again?
>
> Gabor Melis
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Sat Jan 20 22:08:24 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sat, 20 Jan 2007 17:08:24 -0500
Subject: [elephant-devel] Serialization
In-Reply-To: <20061122013435.GL24579@bateleur.arcanes.fr.eu.org>
References: <4561C218.7050606@csail.mit.edu> <87hcwsjrse.fsf@memetrics.com>
	<20061122000409.GJ24579@bateleur.arcanes.fr.eu.org>
	<3B0AF974-F20E-4A81-8BCD-89E78F210C8E@csail.mit.edu>
	<20061122013435.GL24579@bateleur.arcanes.fr.eu.org>
Message-ID: <8F54AA4E-2C57-44DA-BE20-FA26075D03FE@csail.mit.edu>

I just checked in the optimize-option patch to 0.6.1 HEAD.

Ian

On Nov 21, 2006, at 8:34 PM, Pierre THIERRY wrote:

> Scribit Ian Eslick dies 21/11/2006 hora 19:18:
>> I have no objections to the  proposal you sent out earlier (i.e.,
>> extra *feature* and reader  conditionals on optimizations)
>
> I made the optimize form conditional everywhere it appears, here is  
> the
> patch for HEAD.
>
> Quickly,
> Pierre
> -- 
> nowhere.man at levallois.eu.org
> OpenPGP 0xD9D50D8A
> <optimize-option.diff>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070120/ba1a8061/attachment.sig>

From eslick at csail.mit.edu  Sat Jan 20 22:08:48 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sat, 20 Jan 2007 17:08:48 -0500
Subject: [elephant-devel] Removing warnings
In-Reply-To: <20061216091023.GQ6899@bateleur.arcanes.fr.eu.org>
References: <20061216091023.GQ6899@bateleur.arcanes.fr.eu.org>
Message-ID: <C0C43918-14DF-4FB3-AEA5-BE6EFCD59FA8@csail.mit.edu>

Did you ever send this?  I'd be happy to promote it.

Ian

On Dec 16, 2006, at 4:10 AM, Pierre THIERRY wrote:

> Would you mind if a provide a patch that only add at least some  
> generic
> functions, with the only goal to remove the associated style-warnings?
>
> Quickly,
> Pierre
> -- 
> nowhere.man at levallois.eu.org
> OpenPGP 0xD9D50D8A
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070120/0974c8fd/attachment.sig>

From eslick at csail.mit.edu  Sat Jan 20 22:09:21 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sat, 20 Jan 2007 17:09:21 -0500
Subject: [elephant-devel] checkpoint
In-Reply-To: <200701201957.52833.mega@retes.hu>
References: <200701201957.52833.mega@retes.hu>
Message-ID: <BA4DAC5F-09AD-460B-A6F4-F0C988237B5C@csail.mit.edu>

I added this to the 0.6.1 HEAD, along with the optimize-option  
submitted by Pierre.  Checkin to arrive shortly.

Ian

On Jan 20, 2007, at 1:57 PM, G?bor Melis wrote:

> The attached patch adds support DB_ENV.txn_checkpoint that's useful if
> one wants to schedule checkpointing from lisp.
>
> Regards, Gabor Melis
> <checkpoint.diff>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Sat Jan 20 22:18:28 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sat, 20 Jan 2007 17:18:28 -0500
Subject: [elephant-devel] defaults
In-Reply-To: <200701201955.51390.mega@retes.hu>
References: <200701201955.51390.mega@retes.hu>
Message-ID: <753998FB-A7CB-4A39-9504-36308B47AA5E@csail.mit.edu>

I did not promote the defaults as I've already cleaned this up quite  
a bit for 0.6.1.  There is now a config.sexp file that should (or  
will) be copied to my-config.sexp that can be altered to bind all the  
site-specific and user-specific options such as paths to libraries  
and include files, optimization options, etc.

This defaults to not linking the pthread library, but this is easily  
altered by the user.

Anyone reading the list can safely apply this to a local copy of  
0.6.0 if they like.

Ian

On Jan 20, 2007, at 1:55 PM, G?bor Melis wrote:

> Hello
>
> The attached patch (against 0.6.0) does two things:
>
> * changes the default locations of the db4.3 on linux to what I guess
> should match what most users have and the filesystem hierarchy  
> standard
>
> * guards against loading the pthreads library on threaded sbcl  
> where the
> right version is already loaded and it crashes sbcl
>
> Cheers, Gabor Melis
> <default.diff>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From mega at retes.hu  Sat Jan 20 22:57:45 2007
From: mega at retes.hu (=?iso-8859-1?q?G=E1bor_Melis?=)
Date: Sat, 20 Jan 2007 23:57:45 +0100
Subject: [elephant-devel] run-elephant-thread
In-Reply-To: <A9498CEB-77D1-4239-8D8E-875373A4FBDF@csail.mit.edu>
References: <200701202042.22816.mega@retes.hu>
	<A9498CEB-77D1-4239-8D8E-875373A4FBDF@csail.mit.edu>
Message-ID: <200701202357.46270.mega@retes.hu>

On Saturday 20 January 2007 22:54, Ian Eslick wrote:
> There should be an e-mail about this in the archive somewhere, but to
> summarize...
>
> I think that requiring client code to worry about wrapping every
> thread in run-elephant-thread is unrealistic, so that interface is
> now deprecated.  The reason for this is simple, when you are using a
> multithreaded web server the main thread launches client threads
> inside the server code and there is no way to wrap it in an elephant
> thread without modifying the web server and this is unrealistic.

For my web application it is entirely realistic: the webserver need not 
be modified, elephant using code was wrapped in run-elephant-thread - 
that only bound a few specials - within the _client code_ and all was 
well.

> A Thread-Safe Serializer:
>
> The answer to this problem is a little more important than the
> others.  The serializer is called all the time and is a performance
> critical part of the system.  The serializer is thread safe except
> for it's use of a hash table used to detect circular data
> structures.  I've used elephant in a multi-threaded setting for quite
> some time without worrying about the serializer because in practice I
> would never hit the case where an object I was serializing would
> accidentally lookup a duplicate object/id in the circularity hash.
> The way to avoid this case in general is to have a queue of hash
> tables that each thread can grab when it wants to serialize an object
> (you don't want to allocate a hash table in an inner loop - reuse by
> clearing is also costly, but about 50% as much).  So only this queue
> needs to be protected.
>
> I tried using standard locks in the various EXCL-like extended
> packages but the performance is atrocious for frequently called
> routines like serialize.  Instead, I use without-interrupts (a common
> lisp primitive) to block interrupts for the duration of a vector-pop
> command that grabs a new hash.  This gives much better performance.
> The only other variables that need to be so protected are:

Without-interrupts on allegro prohibits multitasking. It is between hard 
and impossible to implement on a true multithreaded lisp that runs on 
multiple cpus (and it kills performance anyway). You won't find an 
equivalent for without-interrupts in sbcl. Actually there is a 
without-interrupts macro in sbcl, but it does something else.

So without-interrupts may work fine on allegro, but on sbcl you need to 
use either mutexes, spinlocks or use thread local specials.

> Thread Safety in Backends:
>
> I'm pretty sure that the current behavior of BDB is thread-safe.  I
> researched this earlier, but only remember that I concluded it was
> safe so if anyone remembers the details feel free to contradict me.
>
> A quick Google investigation says that CL-SQL requires, at a minimum,
> that each thread have it's own connection object to be thread safe,
> so each thread needs to reconnect to the cl-sql database plus have a
> thread-local clsql:*default-database* binding.  (I don't think this
> works for SQLite though)
>
> These are pretty easy and will be handled in my next checkin:
> -------------------------------------------------------------
>
> Global variables (infrequently written):
> *elephant-controller-init*
> *dbconnection-spec*
>
> Store-controller slots that need infrequent write-protection:
> - instance-cache
> - symbol-cache (0.6.1+)
>
>
> The following elephant variables are a little tricky:
> -----------------------------------------------------
>
> Thread-local global vars (frequently accessed):
> *store-controller* (if different threads use different controllers)
> *transaction-stack*
> *current-transaction*
> *resourced-byte-spec*
> (errno handling in uffi?)
>
> Deprecated thread-local vars:
> *auto-commit* (BDB 4.4 no longer pays attention to auto-commit
> arguments so we can remove this from elephant)
>
> 1) You can use the macro with-elephant-variables in 0.6.1 to create
> new, thread-local dynamic bindings of the above variables, but that
> is still a manual solution for when you have access to the thread
> creation code and can create thread-local specials.

As I said above one only needs to be in control somewhere above in the 
dynamic context of any elephant usage. It's error prone but doable.

> 2) A more consequential is to excise required dependency on these
> variables entirely.  The implication of this is a potentially
> significant API change where an application can always provide the
> store controller on calls to collection accessors, cursor operations,
> etc and it defaults to *store-controller* for environments where
> there is only one store or where the user is managing the binding of
> *store-controller* in each thread.  I think this is already
> accommodated in much of the API, but I haven't investigated this to
> see how much work it is.
>
> We can further require that all transactions be wrapped in 'with-
> elephant-transaction' so that the appropriate specials are

Do you mean with-elephant-variables?

> dynamically bound within the stack.  I think this actually would be
> pretty easy.  We could document the internals of with-elephant-
> transaction for anyone who wants to do something sophisticated and is
> willing to manage the thread issues themselves.

Sounds reasonable to me. We really don't want multiple threads to use 
the same values thus the global value of some specials should never be 
used. This situation comes up often and there is a patch on sbcl-devel 
that implements thread local vars that are much like defvars that 
cannot have a global value. This is a side note only as elephant needs 
a portable solution.

> Does anyone have a better suggestion here?  For example is there a
> portability layer that can detect the current thread ID and use that
> to index the default global values?

No, I think what you described above is better and portable. There is 
already a mechanism for thread local storage is multithreaded lisps and 
that should be used.

> Regards,
> Ian

Thank you for the detailed answer.

Gabor


From nowhere.man at levallois.eu.org  Sun Jan 21 03:17:00 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Sun, 21 Jan 2007 04:17:00 +0100
Subject: [elephant-devel] defaults
In-Reply-To: <200701201955.51390.mega@retes.hu>
References: <200701201955.51390.mega@retes.hu>
Message-ID: <20070121031700.GD11824@bateleur.arcanes.fr.eu.org>

Scribit G?bor Melis dies 20/01/2007 hora 19:55:
> * guards against loading the pthreads library on threaded sbcl where
> the right version is already loaded and it crashes sbcl

I always used a threaded SBCL and did not notice any crash, are you sure
it's the problem?

Curiously,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070121/11b6544e/attachment.sig>

From nowhere.man at levallois.eu.org  Sun Jan 21 03:23:16 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Sun, 21 Jan 2007 04:23:16 +0100
Subject: [elephant-devel] run-elephant-thread
In-Reply-To: <1169323600.4833.11.camel@localhost.localdomain>
References: <200701202042.22816.mega@retes.hu>
	<1169323600.4833.11.camel@localhost.localdomain>
Message-ID: <20070121032316.GE11824@bateleur.arcanes.fr.eu.org>

Scribit Robert L. Read dies 20/01/2007 hora 14:06:
> I personally solve problem by using SBCL mutexes around the
> serialize/deserialize code, but I have not committed that code, since
> it is highly SBCL-specific, and in fact I don't know of a good way to
> do locking portably.

I used the bordeux-threads to do that, I attach the patch I personnaly
used. It works like a charm as far as I can see, but as I'm not that
familiar with Elephant's internals, I can't tell if everything is indeed
correctly protected. My applications using it were only put under
moderate workload.

Avoid globals is a Good Thing anyway, and hopefully 0.6.1 will be both
thead-safe and efficient (the lock solution adds a major bottleneck...).

Portably,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mutex.diff
Type: text/x-diff
Size: 4607 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070121/e48e9bda/attachment.diff>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070121/e48e9bda/attachment.sig>

From nowhere.man at levallois.eu.org  Sun Jan 21 03:31:48 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Sun, 21 Jan 2007 04:31:48 +0100
Subject: [elephant-devel] run-elephant-thread
In-Reply-To: <A9498CEB-77D1-4239-8D8E-875373A4FBDF@csail.mit.edu>
References: <200701202042.22816.mega@retes.hu>
	<A9498CEB-77D1-4239-8D8E-875373A4FBDF@csail.mit.edu>
Message-ID: <20070121033148.GF11824@bateleur.arcanes.fr.eu.org>

Scribit Ian Eslick dies 20/01/2007 hora 16:54:
> Does anyone have a better suggestion here?  For example is there a
> portability layer that can detect the current thread ID and use that
> to index the default global values?

http://trac.common-lisp.net/bordeaux-threads/wiki/ApiDocumentation#currentthread

In bordeaux-threads, threads are objects, so you can use them as
indexes.

Quickly,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070121/f83a167d/attachment.sig>

From nowhere.man at levallois.eu.org  Sun Jan 21 03:36:32 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Sun, 21 Jan 2007 04:36:32 +0100
Subject: [elephant-devel] Version 0.5.0 vs. 0.6.0
In-Reply-To: <83BF6578-3502-438E-9A89-051022F6448B@csail.mit.edu>
References: <AEE70B56-B351-40B6-B775-754368AC6B47@csail.mit.edu>
	<20061125151144.GB19309@bateleur.arcanes.fr.eu.org>
	<E2CBEB53-4806-47FC-8567-76ED5ED283A6@csail.mit.edu>
	<20061125185645.GA29568@bateleur.arcanes.fr.eu.org>
	<83BF6578-3502-438E-9A89-051022F6448B@csail.mit.edu>
Message-ID: <20070121033632.GG11824@bateleur.arcanes.fr.eu.org>

Scribit Ian Eslick dies 25/11/2006 hora 14:44:
> Trac and SVN

About the migration to trac: maybe everyone doesn't know, but I
discovered that it can be used with pretty any VCS widely used. I
personnally decided myself to switch to Mercurial and it was trivially
easy to add Mercurial support to it. There is also a darcs plugin, of
course.

I don't know if it's also possible on the common-lisp.net's trac
installation. I suppose they wouldn't mind add at least darcs support,
as it is popular among lispers.

Quickly,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070121/31b163a7/attachment.sig>

From mega at retes.hu  Sun Jan 21 09:00:37 2007
From: mega at retes.hu (=?utf-8?q?G=C3=A1bor_Melis?=)
Date: Sun, 21 Jan 2007 10:00:37 +0100
Subject: [elephant-devel] defaults
In-Reply-To: <20070121031700.GD11824@bateleur.arcanes.fr.eu.org>
References: <200701201955.51390.mega@retes.hu>
	<20070121031700.GD11824@bateleur.arcanes.fr.eu.org>
Message-ID: <200701211000.37288.mega@retes.hu>

On Sunday 21 January 2007 04:17, Pierre THIERRY wrote:
> Scribit G?bor Melis dies 20/01/2007 hora 19:55:
> > * guards against loading the pthreads library on threaded sbcl
> > where the right version is already loaded and it crashes sbcl
>
> I always used a threaded SBCL and did not notice any crash, are you
> sure it's the problem?

Yes. I guess it depends on the installation and on my system (plain 
debian etch) it loaded the wrong version of the lib. Anyway it is 
guaranteed to be loaded with sb-thread.

>
> Curiously,
> Pierre


From eslick at csail.mit.edu  Sun Jan 21 22:21:02 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sun, 21 Jan 2007 17:21:02 -0500
Subject: [elephant-devel] BTREE Sorting on Symbol Strings?
Message-ID: <5E17E008-7D97-4D9D-B62B-73E1546B018A@csail.mit.edu>

How often do users of Elephant rely on BTrees where the symbols are  
ordered according to the symbol's name?

Would it be terribly inconvenient to you if you had to convert symbol  
keys to strings to get an alphabetical ordering, but were still  
assured of contiguity of identical symbol keys in secondary btrees?

The argument is that we serialize symbols to strings all the time  
(slot access, etc) and this engenders a great deal of overhead.  Most  
symbol serialization is highly redundant and can be factored out by  
assigning a persistent ID to each symbol as we do with persistent  
objects.  This results in significantly less disk space (a constant  
vs. N*char_width bits for every slot value), reduced IO bandwidth  
(and increased locality), and less serialization/deserialization time.

The downside is that the C function passed to BDB which is used to  
compare two strings so that the BTree is ordered does not have access  
to the persistent table.  Thus we can't order strings according to  
their characters, but only according to their ID which means a random  
order.  Of course symbols will be identical to themselves and so will  
be grouped together in duplicate indices.

Sorting according to the characters of the symbol may be possible,  
but there are a number of implications that require some thinking  
about and I want to put this off for now.

As this is a user configurable option (my-config.sexp) and will be  
enabled by default I don't think there is any harm in promoting this.

Comments?

Thank you,
Ian


From eslick at csail.mit.edu  Mon Jan 22 01:55:42 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Sun, 21 Jan 2007 20:55:42 -0500
Subject: [elephant-devel] BTREE Sorting on Symbol Strings?
In-Reply-To: <5E17E008-7D97-4D9D-B62B-73E1546B018A@csail.mit.edu>
References: <5E17E008-7D97-4D9D-B62B-73E1546B018A@csail.mit.edu>
Message-ID: <EB71DEB9-0333-4C6C-B780-16422C33D518@csail.mit.edu>

I should have said disabled by default...

On Jan 21, 2007, at 5:21 PM, Ian Eslick wrote:

> How often do users of Elephant rely on BTrees where the symbols are  
> ordered according to the symbol's name?
>
> Would it be terribly inconvenient to you if you had to convert  
> symbol keys to strings to get an alphabetical ordering, but were  
> still assured of contiguity of identical symbol keys in secondary  
> btrees?
>
> The argument is that we serialize symbols to strings all the time  
> (slot access, etc) and this engenders a great deal of overhead.   
> Most symbol serialization is highly redundant and can be factored  
> out by assigning a persistent ID to each symbol as we do with  
> persistent objects.  This results in significantly less disk space  
> (a constant vs. N*char_width bits for every slot value), reduced IO  
> bandwidth (and increased locality), and less serialization/ 
> deserialization time.
>
> The downside is that the C function passed to BDB which is used to  
> compare two strings so that the BTree is ordered does not have  
> access to the persistent table.  Thus we can't order strings  
> according to their characters, but only according to their ID which  
> means a random order.  Of course symbols will be identical to  
> themselves and so will be grouped together in duplicate indices.
>
> Sorting according to the characters of the symbol may be possible,  
> but there are a number of implications that require some thinking  
> about and I want to put this off for now.
>
> As this is a user configurable option (my-config.sexp) and will be  
> enabled by default I don't think there is any harm in promoting this.
>
> Comments?
>
> Thank you,
> Ian
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From nowhere.man at levallois.eu.org  Mon Jan 22 06:42:11 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Mon, 22 Jan 2007 07:42:11 +0100
Subject: [elephant-devel] BTREE Sorting on Symbol Strings?
In-Reply-To: <5E17E008-7D97-4D9D-B62B-73E1546B018A@csail.mit.edu>
References: <5E17E008-7D97-4D9D-B62B-73E1546B018A@csail.mit.edu>
Message-ID: <20070122064210.GI11824@bateleur.arcanes.fr.eu.org>

Scribit Ian Eslick dies 21/01/2007 hora 17:21:
> How often do users of Elephant rely on BTrees where the symbols are
> ordered according to the symbol's name?

That's strange enough for me that I can't think of any use case at first
try...

> Most  symbol serialization is highly redundant and can be factored out
> by  assigning a persistent ID to each symbol as we do with persistent
> objects.

That seems a very desirable optimization indeed.

> As this is a user configurable option (my-config.sexp) and will be
> [disabled] by default I don't think there is any harm in promoting
> this.

I agree, but as I said, I don't see how symbols as strings in BTree
indexes may help.

Quickly,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070122/bb91c991/attachment.sig>

From henrik at evahjelte.com  Mon Jan 22 12:03:51 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Mon, 22 Jan 2007 13:03:51 +0100
Subject: [elephant-devel] BTREE Sorting on Symbol Strings?
In-Reply-To: <5E17E008-7D97-4D9D-B62B-73E1546B018A@csail.mit.edu>
References: <5E17E008-7D97-4D9D-B62B-73E1546B018A@csail.mit.edu>
Message-ID: <1169467431.8265.73.camel@trinidad>

Speaking for the lurkers of this list, it seems like a great idea. 
/Henrik Hjelte




From eslick at csail.mit.edu  Mon Jan 22 16:30:41 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Mon, 22 Jan 2007 11:30:41 -0500
Subject: [elephant-devel] Status of HEAD and 0.6.1
Message-ID: <423110D2-24DC-4147-BCED-E68BBCC69790@csail.mit.edu>

The re-organization for the 0.6.1 feature set has stabilized and the  
current HEAD passes all tests under Allegro 8.0 / OS X / Intel  
against BDB 4.4.

This reorganization is a big step towards several new features to be  
added shortly:
- Support for symbol id's
- Finish support for 64-bit Lisp's
- Clean up support for multi-threading and multiple store controllers

There is a laundry list of minor features planned for 0.6.1.  If  
anyone would like to contribute, now is a good time to jump in while  
the development tree is somewhat stable.  I'll move the source tree  
to SVN and enter outstanding issues into Trac after these features  
are done.

I would also appreciate anyone who wants to run the test suite on  
alternate lisps, architectures and OS's.  In particular, a Win32  
tester is much needed as the new build process doesn't work on  
win32.  I think it's a minimal amount of work.

This check-in includes the following fixes and enhancements:

Major:
x Modularize serializers for easy upgrade
x Implement backend support for symbol-table protocol
x MCL 1.1 unicode support; clean up other lisp support for unicode
x Simplify user-specific configuration parameters using config.sexp  
and my-config.sexp
x Ensure thread safety in buffer-stream allocation!
x Simplify unicode support, database format has a canonical form  
which is translated by lisp
x Remove DB version tags, created deadlock with modular serializers.   
All DB directories now have a file VERSION associated with it.

Minor:
x Diffs for lisp-controlled DB checkpointing (by Gabor Melis)
x Improved optimization options to be more user controlled (Pierre  
Thierry)
x New build interface; remove Makefiles (sans win32), (help from  
elephant-devel, I forget who)
x Think through default *store-controller* vs. explicit parameter  
passing referencing all over the APIs (Enable explicit passing  
everywhere, maintain *store-controller* defaults.  This makes multi- 
threading support simpler.  Users can pass the store controller or  
rely on a global *store-controller*)
x Verify that operations such as indexing are thread safe
x Investigated gensym warnings in berkeley-db.lisp (caused by an FFI  
macro, no harm in it)
x Remove all sleepycat references and use berkeley-db or bdb instead  
(due to Oracle acquisition)
x Remove warnings in libberkeley-db.c

Regards,
Ian


From lists at infoway.net  Mon Jan 22 20:01:11 2007
From: lists at infoway.net (lists at infoway.net)
Date: Mon, 22 Jan 2007 15:01:11 -0500 (EST)
Subject: [elephant-devel] Status of HEAD and 0.6.1
Message-ID: <57266.192.168.1.35.1169496071.webmail@192.168.1.35>

That's great work.

If needed, I can run tests on MacBook Intel with OpenMCL + SBCL with threads compiled in. Just let me know when/where to fetch it from.

Thanks,
Daniel

On Mon, January 22, 2007 11:30 am, Ian Eslick <eslick at csail.mit.edu> said:

> The re-organization for the 0.6.1 feature set has stabilized and the
> current HEAD passes all tests under Allegro 8.0 / OS X / Intel
> against BDB 4.4.
> 
> This reorganization is a big step towards several new features to be
> added shortly:
> - Support for symbol id's
> - Finish support for 64-bit Lisp's
> - Clean up support for multi-threading and multiple store controllers
> 
> There is a laundry list of minor features planned for 0.6.1.  If
> anyone would like to contribute, now is a good time to jump in while
> the development tree is somewhat stable.  I'll move the source tree
> to SVN and enter outstanding issues into Trac after these features
> are done.
> 
> I would also appreciate anyone who wants to run the test suite on
> alternate lisps, architectures and OS's.  In particular, a Win32
> tester is much needed as the new build process doesn't work on
> win32.  I think it's a minimal amount of work.
> 
> This check-in includes the following fixes and enhancements:
> 
> Major:
> x Modularize serializers for easy upgrade
> x Implement backend support for symbol-table protocol
> x MCL 1.1 unicode support; clean up other lisp support for unicode
> x Simplify user-specific configuration parameters using config.sexp
> and my-config.sexp
> x Ensure thread safety in buffer-stream allocation!
> x Simplify unicode support, database format has a canonical form
> which is translated by lisp
> x Remove DB version tags, created deadlock with modular serializers.
> All DB directories now have a file VERSION associated with it.
> 
> Minor:
> x Diffs for lisp-controlled DB checkpointing (by Gabor Melis)
> x Improved optimization options to be more user controlled (Pierre
> Thierry)
> x New build interface; remove Makefiles (sans win32), (help from
> elephant-devel, I forget who)
> x Think through default *store-controller* vs. explicit parameter
> passing referencing all over the APIs (Enable explicit passing
> everywhere, maintain *store-controller* defaults.  This makes multi-
> threading support simpler.  Users can pass the store controller or
> rely on a global *store-controller*)
> x Verify that operations such as indexing are thread safe
> x Investigated gensym warnings in berkeley-db.lisp (caused by an FFI
> macro, no harm in it)
> x Remove all sleepycat references and use berkeley-db or bdb instead
> (due to Oracle acquisition)
> x Remove warnings in libberkeley-db.c
> 
> Regards,
> Ian
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
> 



From read at robertlread.net  Mon Jan 22 20:16:27 2007
From: read at robertlread.net (Robert L. Read)
Date: Mon, 22 Jan 2007 14:16:27 -0600
Subject: [elephant-devel] Status of HEAD and 0.6.1
In-Reply-To: <57266.192.168.1.35.1169496071.webmail@192.168.1.35>
References: <57266.192.168.1.35.1169496071.webmail@192.168.1.35>
Message-ID: <1169496988.24815.90.camel@localhost.localdomain>

I'm beginning testing under SBCL right now; perhaps you should
prioritize testing OpenMCL?

The standard command will get you the head:

cvs -z3 -d :pserver:anonymous:anonymous at common-
lisp.net:/project/elephant/cvsroot co elephant

...until Ian moves us to SVN, which, as I understand it, he has not yet
done.

On Mon, 2007-01-22 at 15:01 -0500, lists at infoway.net wrote:

> That's great work.
> 
> If needed, I can run tests on MacBook Intel with OpenMCL + SBCL with threads compiled in. Just let me know when/where to fetch it from.
> 
> Thanks,
> Daniel
> 
> On Mon, January 22, 2007 11:30 am, Ian Eslick <eslick at csail.mit.edu> said:
> 
> > The re-organization for the 0.6.1 feature set has stabilized and the
> > current HEAD passes all tests under Allegro 8.0 / OS X / Intel
> > against BDB 4.4.
> > 
> > This reorganization is a big step towards several new features to be
> > added shortly:
> > - Support for symbol id's
> > - Finish support for 64-bit Lisp's
> > - Clean up support for multi-threading and multiple store controllers
> > 
> > There is a laundry list of minor features planned for 0.6.1.  If
> > anyone would like to contribute, now is a good time to jump in while
> > the development tree is somewhat stable.  I'll move the source tree
> > to SVN and enter outstanding issues into Trac after these features
> > are done.
> > 
> > I would also appreciate anyone who wants to run the test suite on
> > alternate lisps, architectures and OS's.  In particular, a Win32
> > tester is much needed as the new build process doesn't work on
> > win32.  I think it's a minimal amount of work.
> > 
> > This check-in includes the following fixes and enhancements:
> > 
> > Major:
> > x Modularize serializers for easy upgrade
> > x Implement backend support for symbol-table protocol
> > x MCL 1.1 unicode support; clean up other lisp support for unicode
> > x Simplify user-specific configuration parameters using config.sexp
> > and my-config.sexp
> > x Ensure thread safety in buffer-stream allocation!
> > x Simplify unicode support, database format has a canonical form
> > which is translated by lisp
> > x Remove DB version tags, created deadlock with modular serializers.
> > All DB directories now have a file VERSION associated with it.
> > 
> > Minor:
> > x Diffs for lisp-controlled DB checkpointing (by Gabor Melis)
> > x Improved optimization options to be more user controlled (Pierre
> > Thierry)
> > x New build interface; remove Makefiles (sans win32), (help from
> > elephant-devel, I forget who)
> > x Think through default *store-controller* vs. explicit parameter
> > passing referencing all over the APIs (Enable explicit passing
> > everywhere, maintain *store-controller* defaults.  This makes multi-
> > threading support simpler.  Users can pass the store controller or
> > rely on a global *store-controller*)
> > x Verify that operations such as indexing are thread safe
> > x Investigated gensym warnings in berkeley-db.lisp (caused by an FFI
> > macro, no harm in it)
> > x Remove all sleepycat references and use berkeley-db or bdb instead
> > (due to Oracle acquisition)
> > x Remove warnings in libberkeley-db.c
> > 
> > Regards,
> > Ian
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> > 
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070122/dd1584bb/attachment.html>

From henrik at evahjelte.com  Mon Jan 22 20:50:46 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Mon, 22 Jan 2007 21:50:46 +0100
Subject: [elephant-devel] cvs vs svn vs darcs
Message-ID: <1169499046.8265.137.camel@trinidad>

For those of us that don't know cvs and don't want to learn svn,
here is a way I use to convert elephant cvs to darcs. 
I don't want to start a flamewar, but is there a reason to switch to
subversion rather than darcs? 

There is a way to run Trac with darcs, which I believe is already set up
on common-lisp.net: http://progetti.arstecnica.it/trac+darcs

Now I'll start testing the new code, hope I can help with some issues.

/Henrik Hjelte


Get the tailor python script here:
darcs get http://darcs.arstecnica.it/tailor/

Make a file elephant-cvs-darcs.tailor like below, and
run:
tailor elephant-cvs-darcs.tailor
Just run it again to get the latest cvs changes.

#-----------------------------------------------------------------------
# For syncing cvs elephant to local darcs
[DEFAULT]
verbose = True

[project]
target = darcs:target
start-revision = INITIAL
root-directory = .
state-file = tailor.state
source = cvs:source

[darcs:target]
subdir = darcs

[cvs:source]
module = elephant
repository= :pserver:anonymous:anonymous at common-lisp.net:/project/elephant/cvsroot
subdir = originalcvs





From read at robertlread.net  Mon Jan 22 21:12:23 2007
From: read at robertlread.net (Robert L. Read)
Date: Mon, 22 Jan 2007 15:12:23 -0600
Subject: [elephant-devel] cvs vs svn vs darcs
In-Reply-To: <1169499046.8265.137.camel@trinidad>
References: <1169499046.8265.137.camel@trinidad>
Message-ID: <1169500344.24815.97.camel@localhost.localdomain>

Common-lisp.net, where elephant is hosted, offers SVN as a standard
service, 
and we might have to install and support darcs ourselves.

(Personally, I'll support whatever Ian decides.  I've tried to use Darcs
and subversion,
and can't really see enough of a difference with CVS to matter --- but
then I used CVS
in a job for a long time, so I am quite familiar with it.)

On Mon, 2007-01-22 at 21:50 +0100, Henrik Hjelte wrote:

> For those of us that don't know cvs and don't want to learn svn,
> here is a way I use to convert elephant cvs to darcs. 
> I don't want to start a flamewar, but is there a reason to switch to
> subversion rather than darcs? 
> 
> There is a way to run Trac with darcs, which I believe is already set up
> on common-lisp.net: http://progetti.arstecnica.it/trac+darcs
> 
> Now I'll start testing the new code, hope I can help with some issues.
> 
> /Henrik Hjelte
> 
> 
> Get the tailor python script here:
> darcs get http://darcs.arstecnica.it/tailor/
> 
> Make a file elephant-cvs-darcs.tailor like below, and
> run:
> tailor elephant-cvs-darcs.tailor
> Just run it again to get the latest cvs changes.
> 
> #-----------------------------------------------------------------------
> # For syncing cvs elephant to local darcs
> [DEFAULT]
> verbose = True
> 
> [project]
> target = darcs:target
> start-revision = INITIAL
> root-directory = .
> state-file = tailor.state
> source = cvs:source
> 
> [darcs:target]
> subdir = darcs
> 
> [cvs:source]
> module = elephant
> repository= :pserver:anonymous:anonymous at common-lisp.net:/project/elephant/cvsroot
> subdir = originalcvs
> 
> 
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070122/8565c619/attachment.html>

From eslick at csail.mit.edu  Mon Jan 22 21:09:08 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Mon, 22 Jan 2007 16:09:08 -0500
Subject: [elephant-devel] cvs vs svn vs darcs
In-Reply-To: <1169499046.8265.137.camel@trinidad>
References: <1169499046.8265.137.camel@trinidad>
Message-ID: <EA13547A-8A66-4525-8CF6-2A95FD27E811@csail.mit.edu>

If you want to help with the switch to darcs I'd prefer to use it too  
- anyone else want to weigh in on this?

Ian


On Jan 22, 2007, at 3:50 PM, Henrik Hjelte wrote:

> For those of us that don't know cvs and don't want to learn svn,
> here is a way I use to convert elephant cvs to darcs.
> I don't want to start a flamewar, but is there a reason to switch to
> subversion rather than darcs?
>
> There is a way to run Trac with darcs, which I believe is already  
> set up
> on common-lisp.net: http://progetti.arstecnica.it/trac+darcs
>
> Now I'll start testing the new code, hope I can help with some issues.
>
> /Henrik Hjelte
>
>
> Get the tailor python script here:
> darcs get http://darcs.arstecnica.it/tailor/
>
> Make a file elephant-cvs-darcs.tailor like below, and
> run:
> tailor elephant-cvs-darcs.tailor
> Just run it again to get the latest cvs changes.
>
> #--------------------------------------------------------------------- 
> --
> # For syncing cvs elephant to local darcs
> [DEFAULT]
> verbose = True
>
> [project]
> target = darcs:target
> start-revision = INITIAL
> root-directory = .
> state-file = tailor.state
> source = cvs:source
>
> [darcs:target]
> subdir = darcs
>
> [cvs:source]
> module = elephant
> repository= :pserver:anonymous:anonymous at common-lisp.net:/project/ 
> elephant/cvsroot
> subdir = originalcvs
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From cjstuij at gmail.com  Mon Jan 22 21:36:58 2007
From: cjstuij at gmail.com (Ties Stuij)
Date: Mon, 22 Jan 2007 22:36:58 +0100
Subject: [elephant-devel] cvs vs svn vs darcs
In-Reply-To: <EA13547A-8A66-4525-8CF6-2A95FD27E811@csail.mit.edu>
References: <1169499046.8265.137.camel@trinidad>
	<EA13547A-8A66-4525-8CF6-2A95FD27E811@csail.mit.edu>
Message-ID: <d7ce4ebf0701221336o1e2377ddqc854d22521e7f290@mail.gmail.com>

On 1/22/07, Ian Eslick <eslick at csail.mit.edu> wrote:
> If you want to help with the switch to darcs I'd prefer to use it too
> - anyone else want to weigh in on this?

fwiw, i'm for darcs
Ties


From henrik at evahjelte.com  Mon Jan 22 22:09:23 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Mon, 22 Jan 2007 23:09:23 +0100
Subject: [elephant-devel] bdb-slots.lisp is missing
Message-ID: <1169503763.8265.161.camel@trinidad>

>From cvs, it should be there right?

Because otherwise 69 out of 115 tests fail for me. 

And sure I can help with setting up darcs/trac on common-lisp.net if
darcs gets the vote. I am hhjelte on common-lisp.net

/Henrik Hjelte





From eslick at csail.mit.edu  Mon Jan 22 22:32:19 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Mon, 22 Jan 2007 17:32:19 -0500
Subject: [elephant-devel] bdb-slots.lisp is missing
In-Reply-To: <1169503763.8265.161.camel@trinidad>
References: <1169503763.8265.161.camel@trinidad>
Message-ID: <DEF67B5C-EC42-4698-B930-2CD0BFC70DE7@csail.mit.edu>

added.  i forgot to check out a clean copy and test it - doing so  
now.  -Ian

On Jan 22, 2007, at 5:09 PM, Henrik Hjelte wrote:

>> From cvs, it should be there right?
>
> Because otherwise 69 out of 115 tests fail for me.
>
> And sure I can help with setting up darcs/trac on common-lisp.net if
> darcs gets the vote. I am hhjelte on common-lisp.net
>
> /Henrik Hjelte
>
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From lists at infoway.net  Mon Jan 22 22:38:13 2007
From: lists at infoway.net (lists at infoway.net)
Date: Mon, 22 Jan 2007 17:38:13 -0500 (EST)
Subject: [elephant-devel] bdb-slots.lisp is missing
Message-ID: <59663.192.168.1.71.1169505493.webmail@192.168.1.71>

I just downloaded from CVS and am still getting missing bdb-slots.lisp.

Also, for the record, I my "out-of-the-box" my-config.sexp looks like:

((:berkeley-db-root . "/usr/local/BerkeleyDB.4.4")
 (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.4/include")
 (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.4/lib")
 (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.4/lib/libdb-4.4.dylib")
 (:pthread-lib . nil)
 (:clsql-lib . nil)
 (:fast-symbols . nil))

on SBCL 1.0 for MacBook.

Once I get bdb-slots.lisp I will continue with the tests.

Thanks,
Daniel

On Mon, January 22, 2007 5:32 pm, Ian Eslick <eslick at csail.mit.edu> said:

> added.  i forgot to check out a clean copy and test it - doing so
> now.  -Ian
> 
> On Jan 22, 2007, at 5:09 PM, Henrik Hjelte wrote:
> 
>>> From cvs, it should be there right?
>>
>> Because otherwise 69 out of 115 tests fail for me.
>>
>> And sure I can help with setting up darcs/trac on common-lisp.net if
>> darcs gets the vote. I am hhjelte on common-lisp.net
>>
>> /Henrik Hjelte
>>
>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
> 



From eslick at csail.mit.edu  Mon Jan 22 23:14:42 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Mon, 22 Jan 2007 18:14:42 -0500
Subject: [elephant-devel] bdb-slots.lisp is missing
In-Reply-To: <59663.192.168.1.71.1169505493.webmail@192.168.1.71>
References: <59663.192.168.1.71.1169505493.webmail@192.168.1.71>
Message-ID: <A76D4BE9-2CBF-46AD-89D4-9D2F40153D77@csail.mit.edu>

Try cvs update.  I had the same problem when I checked out a fresh  
copy, but after an update a few minutes ago it downloaded bdb- 
slots.lisp and passed all tests (Allegro 8.0 / Mac OS X / Intel x86)

Ian


On Jan 22, 2007, at 5:38 PM, lists at infoway.net wrote:

> I just downloaded from CVS and am still getting missing bdb- 
> slots.lisp.
>
> Also, for the record, I my "out-of-the-box" my-config.sexp looks like:
>
> ((:berkeley-db-root . "/usr/local/BerkeleyDB.4.4")
>  (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.4/include")
>  (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.4/lib")
>  (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.4/lib/libdb-4.4.dylib")
>  (:pthread-lib . nil)
>  (:clsql-lib . nil)
>  (:fast-symbols . nil))
>
> on SBCL 1.0 for MacBook.
>
> Once I get bdb-slots.lisp I will continue with the tests.
>
> Thanks,
> Daniel
>
> On Mon, January 22, 2007 5:32 pm, Ian Eslick <eslick at csail.mit.edu>  
> said:
>
>> added.  i forgot to check out a clean copy and test it - doing so
>> now.  -Ian
>>
>> On Jan 22, 2007, at 5:09 PM, Henrik Hjelte wrote:
>>
>>>> From cvs, it should be there right?
>>>
>>> Because otherwise 69 out of 115 tests fail for me.
>>>
>>> And sure I can help with setting up darcs/trac on common-lisp.net if
>>> darcs gets the vote. I am hhjelte on common-lisp.net
>>>
>>> /Henrik Hjelte
>>>
>>>
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Mon Jan 22 23:34:49 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Mon, 22 Jan 2007 18:34:49 -0500
Subject: [elephant-devel] bdb-slots.lisp is missing
In-Reply-To: <A76D4BE9-2CBF-46AD-89D4-9D2F40153D77@csail.mit.edu>
References: <59663.192.168.1.71.1169505493.webmail@192.168.1.71>
	<A76D4BE9-2CBF-46AD-89D4-9D2F40153D77@csail.mit.edu>
Message-ID: <129DEEF3-7DCE-41D3-9DAC-9D1AE574ED52@csail.mit.edu>

FYI - The fresh download also passed all tests on SBCL 0.9.17 / Mac  
OS X / Intel.

NOTE: If you are testing multiple lisp implementations, be sure to  
run tests/delscript.sh in the elephant/tests directory prior to  
running a second backend test.

Ian

On Jan 22, 2007, at 6:14 PM, Ian Eslick wrote:

> Try cvs update.  I had the same problem when I checked out a fresh  
> copy, but after an update a few minutes ago it downloaded bdb- 
> slots.lisp and passed all tests (Allegro 8.0 / Mac OS X / Intel x86)
>
> Ian
>
>
> On Jan 22, 2007, at 5:38 PM, lists at infoway.net wrote:
>
>> I just downloaded from CVS and am still getting missing bdb- 
>> slots.lisp.
>>
>> Also, for the record, I my "out-of-the-box" my-config.sexp looks  
>> like:
>>
>> ((:berkeley-db-root . "/usr/local/BerkeleyDB.4.4")
>>  (:berkeley-db-include-dir . "/usr/local/BerkeleyDB.4.4/include")
>>  (:berkeley-db-lib-dir . "/usr/local/BerkeleyDB.4.4/lib")
>>  (:berkeley-db-lib . "/usr/local/BerkeleyDB.4.4/lib/libdb-4.4.dylib")
>>  (:pthread-lib . nil)
>>  (:clsql-lib . nil)
>>  (:fast-symbols . nil))
>>
>> on SBCL 1.0 for MacBook.
>>
>> Once I get bdb-slots.lisp I will continue with the tests.
>>
>> Thanks,
>> Daniel
>>
>> On Mon, January 22, 2007 5:32 pm, Ian Eslick  
>> <eslick at csail.mit.edu> said:
>>
>>> added.  i forgot to check out a clean copy and test it - doing so
>>> now.  -Ian
>>>
>>> On Jan 22, 2007, at 5:09 PM, Henrik Hjelte wrote:
>>>
>>>>> From cvs, it should be there right?
>>>>
>>>> Because otherwise 69 out of 115 tests fail for me.
>>>>
>>>> And sure I can help with setting up darcs/trac on common- 
>>>> lisp.net if
>>>> darcs gets the vote. I am hhjelte on common-lisp.net
>>>>
>>>> /Henrik Hjelte
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> elephant-devel site list
>>>> elephant-devel at common-lisp.net
>>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Mon Jan 22 23:37:42 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Mon, 22 Jan 2007 18:37:42 -0500
Subject: [elephant-devel] Testing
Message-ID: <F0FBE56F-4138-4263-BDD4-2DD27A14E47C@csail.mit.edu>

Test coverage issues:

- Most changes I just made are subsumed by testing the existing  
regressions on different platforms, such as running the regression on  
64-bit machines with a 64-bit lisp.

- An 0.6.0 DB to 0.6.1 DB migration test (to be added) will validate  
the new multiple-serializer interface as well as migration support.

- Unicode support: we need a test that generates UTF-16 and UTF-32  
strings on platforms that support them and ensures that they are  
properly translated (only SBCL supports UTF-32 at present, correct?)

- Thread safety:  any suggestions on how to do this?  I haven't done  
any formal multi-threaded software engineering, so am unfamiliar with  
the current state of the art in testing these systems (other than  
formally).

Optional:
- A non-automated test that different lisps can connect to the same  
database on the same machine successfully.  A test that writes a DB  
and another one that opens and validates the data should be a nice  
sanity check of this.  This may even work across machine  
architectures, but I haven't thought through how machine endianness  
will effect this.

Cheers,
Ian




From nowhere.man at levallois.eu.org  Tue Jan 23 17:17:41 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Tue, 23 Jan 2007 18:17:41 +0100
Subject: [elephant-devel] cvs vs svn vs darcs
In-Reply-To: <EA13547A-8A66-4525-8CF6-2A95FD27E811@csail.mit.edu>
References: <1169499046.8265.137.camel@trinidad>
	<EA13547A-8A66-4525-8CF6-2A95FD27E811@csail.mit.edu>
Message-ID: <20070123171741.GP11824@bateleur.arcanes.fr.eu.org>

Scribit Ian Eslick dies 22/01/2007 hora 16:09:
> If you want to help with the switch to darcs I'd prefer to use it too
> - anyone else want to weigh in on this?

I've now come to prefer DVCS like Darcs or Mercurial over classical like
CVS or Subversion, mostly because of the freedom they offer. I'll prefer
Mercurial, but would be happy with Darcs too.

Quickly,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070123/bef520b9/attachment.sig>

From nowhere.man at levallois.eu.org  Tue Jan 23 17:25:42 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Tue, 23 Jan 2007 18:25:42 +0100
Subject: [elephant-devel] Testing
In-Reply-To: <F0FBE56F-4138-4263-BDD4-2DD27A14E47C@csail.mit.edu>
References: <F0FBE56F-4138-4263-BDD4-2DD27A14E47C@csail.mit.edu>
Message-ID: <20070123172542.GQ11824@bateleur.arcanes.fr.eu.org>

Scribit Ian Eslick dies 22/01/2007 hora 18:37:
> - Most changes I just made are subsumed by testing the existing
> regressions on different platforms, such as running the regression on
> 64-bit machines with a 64-bit lisp.

I'll run the tests under Debian Etch on an amd64. The same box is
running in 32-bit Windows XP, and I have SBCL 1.0 and the last
evaluation version of Lispworks, so i'll try to help making it work
under Windows.

FWIW, I will probably need Windows support, as I'll deploy applications
on Windows systems in the near future, for my customers. And Elephant is
a bliss when it comes to store a complex graph of objects. :-)

> - Thread safety:  any suggestions on how to do this?  I haven't done
> any formal multi-threaded software engineering, so am unfamiliar with
> the current state of the art in testing these systems (other than
> formally).

I can at least give a test that used to fail for me without the locking
patch I just sent: get Elephant to be called by a multi-threaded web
server like Araneida or Hunchentoot and stuff the server with parallel
requests fast (I used Apache's benchmarking tool).

Quickly,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070123/f6a6d3f7/attachment.sig>

From henrik at evahjelte.com  Wed Jan 24 00:42:29 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Wed, 24 Jan 2007 01:42:29 +0100
Subject: [elephant-devel] sbcl-64 tests
Message-ID: <1169599349.8265.229.camel@trinidad>

I have started running the bdb backendtests on latest sbcl, amd64 linux.
I also have got a beta snapshot of openmcl-linux-64 (2006-12-31) running
sometimes. I don't know how stable this beta is, but at least it's good
to run several lisps to find bugs.

When running the backend tests for bdb on sbcl, the first failure was
the FIXNUM test. It serializes/deserializes most-positive-fixnum. Since
most-positive-fixnum was larger than 32bit integers, I somehow suspected
64 bit problems. Which was not the main problem. Instead it was related
to the type checking in the serializer. I have attached a solution in a
diff-file. 

(A parenthesis: My 64-bit adventures gave some code as a sideffect. I
have added 64 bit integers to memutils.c and refactored/obscured it with
a macro that makes the reader and writer functions. I have changed all
references to int to be explict int32 in memutils.lisp and
bdb-controller. I have a half-finished version of bdb-controller with
macros making the cursor movement methods. I'll keep it in the closet if
it's of any use later on. End of parenthesis)

Now almost all backendtests run.
2 out of 115 total tests failed: INDEXING-WIPE-INDEX, INDEXING-TIMING.
Index-timing fails with this error. This is something I have noted
before with my recent experiments with elephant version 6. Suddently
after deleting instances, elephant would sometimes say that the class
was no longer indexed. I think it showed up with both the bdb and the
clsql backend. Has no one else noticed this? Below is a backtrace, you
can see the drop-instances call which I am suspecting.

About multithreading/testing. I volunteer to add some kind of
stress-test with bourdeaux threads later, but now the outstanding bug is
a blocker.

Speaking of multithreading: since few Lisp:s use native threads, and
writing thread-safe code can be difficult, in my opinion taking
advantage of multi-core processors is most easily done with spreading
processes. what about multiprocessing? I mean parallel Lisp instances,
on the same or different computers, writing and reading to the same
database? Is elephant supposed to work in that kind of environment. I
suppose the clsql backend does, but what about bdb? 

That's all for now, it's in the middle of the night for me, got to count
some sheep.
/Henrik Hjelte


Class #<PERSISTENT-METACLASS STRESS-INDEX> is not an indexed class
   [Condition of type SIMPLE-ERROR]

Restarts:
 0: [ABORT] Return to SLIME's top level.
 1: [TERMINATE-THREAD] Terminate this thread (#<THREAD
"worker" {1004E89801}>)

Backtrace:
  0: ((SB-PCL::FAST-METHOD FIND-CLASS-INDEX (PERSISTENT-METACLASS))
      #<unused argument>
      #<unused argument>
      #<PERSISTENT-METACLASS STRESS-INDEX>
      :SC
      NIL
      :ERRORP
      T)
  1: ((LAMBDA ()))
  2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
       (DB-BDB::BDB-STORE-CONTROLLER #1="#<...>" . #1#))
      #<unused argument>
      #<unused argument>
      #<DB-BDB::BDB-STORE-CONTROLLER {1004E9A2D1}>
      #<CLOSURE (LAMBDA #) {10049CE119}>
      :TRANSACTION
      NIL
      :ENVIRONMENT
      NIL ..)
  3: (DROP-INSTANCES
      (#<STRESS-INDEX {10049CBB51}> #<STRESS-INDEX {10049CA331}>






-------------- next part --------------
Tue Jan 23 23:14:09 CET 2007  Henrik Hjelte <henrik at evahjelte.com>
  * Make most-negative and positive fixnum work on 64 bit sbcl
  Looks like a bigger change than it is because of indentation change.
Tue Jan 23 12:39:21 CET 2007  Henrik Hjelte <henrik at evahjelte.com>
  * openmcl complained about importing something that doesn't exist
Tue Jan 23 11:57:22 CET 2007  Henrik Hjelte <henrik at evahjelte.com>
  * openmcl detected that spec is a list starting with :bdb rather than a string or pathname
Mon Jan 22 22:23:37 CET 2007  Henrik Hjelte <henrik at evahjelte.com>
  * two small compiler warnings
diff -rN -u old-elephant/elephant.asd new-elephant/elephant.asd
--- old-elephant/elephant.asd	2007-01-24 00:26:46.318133014 +0100
+++ new-elephant/elephant.asd	2007-01-24 00:26:46.358133668 +0100
@@ -117,6 +117,7 @@
    "-lm"))
 
 (defmethod compiler-options ((compiler (eql :msvc)) (c elephant-c-source) &key input-file output-file)
+  (declare (ignorable c input-file output-file))
   (error "MSVC compiler option not supported yet"))
 
 ;; LOAD
diff -rN -u old-elephant/src/elephant/controller.lisp new-elephant/src/elephant/controller.lisp
--- old-elephant/src/elephant/controller.lisp	2007-01-24 00:26:46.314132948 +0100
+++ new-elephant/src/elephant/controller.lisp	2007-01-24 00:26:46.334133276 +0100
@@ -138,7 +138,7 @@
 ;;
 
 (defclass store-controller ()
-  ((spec :type (or pathname string (simple-array character))
+  ((spec :type list
 	 :accessor controller-spec
 	 :initarg :spec
 	 :documentation "Backend create functions should pass in :spec during make-instance")
diff -rN -u old-elephant/src/elephant/serializer1.lisp new-elephant/src/elephant/serializer1.lisp
--- old-elephant/src/elephant/serializer1.lisp	2007-01-24 00:26:46.314132948 +0100
+++ new-elephant/src/elephant/serializer1.lisp	2007-01-24 00:26:46.338133341 +0100
@@ -19,7 +19,6 @@
 (defpackage :elephant-serializer1
   (:use :cl :elephant :elephant-memutil)
   (:import-from :elephant 
-		*resourced-byte-spec*
 		get-cached-instance
 		slot-definition-allocation
 		slot-definition-name
diff -rN -u old-elephant/src/elephant/serializer2.lisp new-elephant/src/elephant/serializer2.lisp
--- old-elephant/src/elephant/serializer2.lisp	2007-01-24 00:26:46.310132883 +0100
+++ new-elephant/src/elephant/serializer2.lisp	2007-01-24 00:26:46.342133406 +0100
@@ -20,7 +20,6 @@
   (:use :cl :elephant :elephant-memutil)
   (:import-from :elephant 
 		*circularity-initial-hash-size*
-		*resourced-byte-spec*
 		get-cached-instance
 		controller-symbol-cache 
 		controller-symbol-id-cache
@@ -77,6 +76,11 @@
 (defconstant +struct+               20)
 (defconstant +class+                21)
 
+;; Numerical constants
+(defconstant +most-positive-fixnum+ 22)
+(defconstant +most-negative-fixnum+ 23)
+
+
 (defconstant +nil+                  #x3F)
 
 ;; Arrays
@@ -158,9 +162,6 @@
 	      (serialize-symbol frob bs sc))
 	     (string
 	      (serialize-string frob bs))
-	     ((integer #.most-negative-fixnum #.most-positive-fixnum)
-	      (buffer-write-byte +fixnum32+ bs)
-	      (buffer-write-int frob bs))
 	     (null
 	      (buffer-write-byte +nil+ bs))
 	     (persistent
@@ -197,25 +198,31 @@
 			(declare (dynamic-extent svs))
 			(%serialize (/ (length svs) 2))
 			(loop for item in svs
-			   do (%serialize item)))))))
+                              do (%serialize item)))))))
 	     (integer
-	      (let* ((num (abs frob))
-		     (word-size (ceiling (/ (integer-length num) 32)))
-		     (needed (* word-size 4)))
-		(declare (type fixnum word-size needed))
-		(if (< frob 0) 
-		    (buffer-write-byte +negative-bignum+ bs)
-		    (buffer-write-byte +positive-bignum+ bs))
-		(buffer-write-int needed bs)
-		(loop for i fixnum from 0 below word-size 
-		   ;; this ldb is consing on CMUCL!
-		   ;; there is an OpenMCL function which should work 
-		   ;; and non-cons
-		   do
-		   #+(or cmu sbcl)
-		   (buffer-write-uint (ldb (int-byte-spec i) num) bs) ;; (%bignum-ref num i) bs)
-		   #+(or allegro lispworks openmcl)
-		   (buffer-write-uint (ldb (int-byte-spec i) num) bs))))
+              (cond
+                ((= frob most-positive-fixnum)
+                 (buffer-write-byte +most-positive-fixnum+ bs))
+                ((= frob most-negative-fixnum)
+                 (buffer-write-byte +most-negative-fixnum+ bs))
+                (t
+                 (let* ((num (abs frob))
+                        (word-size (ceiling (/ (integer-length num) 32)))
+                        (needed (* word-size 4)))
+                   (declare (type fixnum word-size needed))
+                   (if (< frob 0) 
+                       (buffer-write-byte +negative-bignum+ bs)
+                       (buffer-write-byte +positive-bignum+ bs))
+                   (buffer-write-int needed bs)
+                   (loop for i fixnum from 0 below word-size 
+                         ;; this ldb is consing on CMUCL!
+                         ;; there is an OpenMCL function which should work 
+                         ;; and non-cons
+                         do
+                         #+(or cmu sbcl)
+                         (buffer-write-uint (ldb (int-byte-spec i) num) bs) ;; (%bignum-ref num i) bs)
+                         #+(or allegro lispworks openmcl)
+                         (buffer-write-uint (ldb (int-byte-spec i) num) bs))))))
 	     (rational
 	      (buffer-write-byte +rational+ bs)
 	      (%serialize (numerator frob))
@@ -251,10 +258,10 @@
 		      (%serialize (hash-table-rehash-threshold frob))
 		      (%serialize (hash-table-count frob))
 		      (loop for key being the hash-key of frob
-			 using (hash-value value)
-			 do 
-			 (%serialize key)
-			 (%serialize value))))))
+                            using (hash-value value)
+                            do 
+                            (%serialize key)
+                            (%serialize value))))))
 	     ;; 	   (structure-object 
 	     ;; 	    (buffer-write-byte +struct+ bs)
 	     ;; 	    (let ((idp (gethash frob *circularity-hash*)))
@@ -286,17 +293,17 @@
 		      (let ((rank (array-rank frob)))
 			(buffer-write-int rank bs)
 			(loop for i fixnum from 0 below rank
-			   do (buffer-write-int (array-dimension frob i) 
-						bs)))
+                              do (buffer-write-int (array-dimension frob i) 
+                                                   bs)))
 		      (when (array-has-fill-pointer-p frob)
 			(buffer-write-int (fill-pointer frob) bs))
 		      (loop for i fixnum from 0 below (array-total-size frob)
-			 do
-			 (%serialize (row-major-aref frob i)))))))
+                            do
+                            (%serialize (row-major-aref frob i)))))))
 	     )))
-    (%serialize frob)
-    (release-circularity-hash *circularity-hash*)
-    bs)))
+      (%serialize frob)
+      (release-circularity-hash *circularity-hash*)
+      bs)))
 
 (defun slots-and-values (o)
   (loop for sd in (compute-slots (class-of o))
@@ -329,6 +336,10 @@
 	   (cond
 	     ((= tag +fixnum32+) 
 	      (buffer-read-fixnum bs))
+             ((= tag +most-positive-fixnum+) 
+	      most-positive-fixnum)
+             ((= tag +most-negative-fixnum+)
+	      most-negative-fixnum)             
 	     ((= tag +nil+) nil)
 	     ((= tag +utf8-string+)
 	      (deserialize-string :utf8 bs))
@@ -378,7 +389,7 @@
 	     ((= tag +hash-table+)
 	      (let* ((id (buffer-read-fixnum bs))
 		     (maybe-hash (lookup-id id)))
-		(declare (dynamic-extent id maybe-cons)
+		(declare (dynamic-extent id maybe-hash)
 			 (type fixnum id))
 		(if maybe-hash maybe-hash
 		    (let ((h (make-hash-table :test (%deserialize bs)
diff -rN -u old-elephant/src/elephant/serializer2-locks.lisp new-elephant/src/elephant/serializer2-locks.lisp
--- old-elephant/src/elephant/serializer2-locks.lisp	2007-01-24 00:26:46.314132948 +0100
+++ new-elephant/src/elephant/serializer2-locks.lisp	2007-01-24 00:26:46.342133406 +0100
@@ -20,7 +20,8 @@
   (:use :cl :elephant :elephant-memutil)
   (:import-from :elephant 
 		*circularity-initial-hash-size*
-		*resourced-byte-spec*
+		#+(or cmu sbcl allegro)
+                *resourced-byte-spec*
 		get-cached-instance
 		slot-definition-allocation
 		slot-definition-name
diff -rN -u old-elephant/src/memutil/memutil.lisp new-elephant/src/memutil/memutil.lisp
--- old-elephant/src/memutil/memutil.lisp	2007-01-24 00:26:46.318133014 +0100
+++ new-elephant/src/memutil/memutil.lisp	2007-01-24 00:26:46.346133472 +0100
@@ -82,7 +82,7 @@
        (length :int))
     :returning :void))
 
-(eval-when (compile)
+(eval-when (:compile-toplevel)
   (declaim 
    #-elephant-without-optimize (optimize (speed 3) (safety 1) (space 0) (debug 0))
    (inline read-int read-uint read-float read-double 


From franks-muc at web.de  Wed Jan 24 00:48:36 2007
From: franks-muc at web.de (franks-muc at web.de)
Date: Wed, 24 Jan 2007 01:48:36 +0100
Subject: [elephant-devel] Testing
Message-ID: <379713322@web.de>


I tried the new elephant on winXP. 

1. Allegro8 trial: 

Error: In :IMPORT list, the symbol "ARRAY-TYPE-FROM-BYTE" not found in package #<The ELEPHANT package>
[condition type: PACKAGE-ERROR]

when loading 
c:\lisp\binaries\allegro-a8.0-mswindows-x86\lisp\libraries\elephant\src\elephant\serializer1.fasl

2. LispWorks 5 pro

in file memutil.lisp

This creates an error:

(def-foreign-type array-or-pointer-char
      #+allegro (:array :char)
      #+(or cmu sbcl scl openmcl) (* :char))

changed 2nd line to:
      #+(or allegro lispworks) (:array :char)
don't know whether it is the correct selection

Then:

Compilation aborted due to error between functions:
  Package SB-KERNEL not found.

when compiling this: 
;; A non-back-compatible change was made in SBCL 8 moving to SBCL 9,
;; in that the function copy-from-system-area disappeared.
;; This code is an attempt to allow compilation under bothe SBCL 8 and SBCL 9.
;; Thanks to Juho Snellman for this idiom.
(eval-when (:compile-toplevel)
  (defun new-style-copy-p ()
    (if (find-symbol "COPY-UB8-FROM-SYSTEM-AREA" "SB-KERNEL") 
	'(:and) 
	'(:or)))
  )

I then commented out all forms depending on
;#+#.(elephant-memutil::new-style-copy-p) and 
;#-#.(elephant-memutil::new-style-copy-p)

in file package lisp

Compilation aborted due to error between functions:
  Duplicated names in "ELEPHANT" defpackage: "ELE".

I replaced 
 (:nicknames ele :ele)
with
 (:nicknames :ele)
  
In file metaclasses.lisp 

**++++ Error in (DEFCLASS ELEPHANT:PERSISTENT): 
  Defining function :DBCN-SPC-PST visible from package KEYWORD.

The accessor function stars with colon. I deleted the colon.

In file classes.lisp

Error
Defining (METHOD ENSURE-CLASS-USING-CLASS :AROUND ((EQL NIL) T)) visible from packages COMMON-LISP, CLOS.

No idea what to do. 


3. I was able to use elephant 6.0 with ACL 7 trial

wherein I used cygwin to produce the dll s, and I copied the dll, with a newer date than the source, into the 
directory with binaries to prevent recompilation started with run-shell-command from asdf, which does not seem 
to work on windows. 

This is the script for libmemutil.dll:

gcc -mno-cygwin -mwindows -std=c99 -c libmemutil.c
dlltool -z libmeutil.def --export-all-symbols -e exports.o -l libmemutil.lib libmemutil.o
gcc -shared -mno-cygwin -mwindows libmemutil.o exports.o -o libmemutil.dll

And this is the script for libsleepycat.dll:

gcc -mno-cygwin -mwindows -c -Wall -std=c99 -L/c/DB/Berkeley\ DB\ 4.4.20/lib/ -I/c/DB/Berkeley\ DB\ 4.4.20/include/ libsleepycat.c
dlltool -z libsleepycat.def --export-all-symbols -e exports.o -l libsleepycat.lib libsleepycat.o
gcc -shared -mno-cygwin -mwindows -L/c/DB/Berkeley\ DB\ 4.4.20/bin/ -llibdb44 libsleepycat.o exports.o -o libsleepycat.dll

(I don't know how to use variables in shell scripts)


4. I will try again 

_______________________________________________________________________
Viren-Scan f?r Ihren PC! Jetzt f?r jeden. Sofort, online und kostenlos.
Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=022222



From eslick at csail.mit.edu  Wed Jan 24 01:00:01 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Tue, 23 Jan 2007 20:00:01 -0500
Subject: [elephant-devel] Testing
In-Reply-To: <379713322@web.de>
References: <379713322@web.de>
Message-ID: <BB146B13-345A-4803-8994-35AB79E59A93@csail.mit.edu>

Thanks to everyone who is doing testing.  We'll take the feedback  
from these runs and clean things up a bit before asking for another  
round of testing.  I think we'll accelerate the 0.6.1 release so  
there is a definite target to test for.  The remaining features:

Serializer efficiency improvement (SQL & BDB)
Native 64-bit support
Simplify multi-threading and multi-store operations
Ensure migration works between 0.6.0 and 0.6.1

I'm hoping to be ready with an alpha by the first week of february.   
Anyone who wants to help with testing in the meantime just keep an  
eye on the list, I'll announce features as they appear ready for a  
broader testing.  Anyone who wants to help with a feature, just let  
me know so we can coordinate our efforts.

For the serializer, I think that Robert and I are most of the way  
there.  Code reviews and comments are fine, but hold off until next  
week.

Ian

On Jan 23, 2007, at 7:48 PM, franks-muc at web.de wrote:

>
> I tried the new elephant on winXP.
>
> 1. Allegro8 trial:
>
> Error: In :IMPORT list, the symbol "ARRAY-TYPE-FROM-BYTE" not found  
> in package #<The ELEPHANT package>
> [condition type: PACKAGE-ERROR]
>
> when loading
> c:\lisp\binaries\allegro-a8.0-mswindows-x86\lisp\libraries\elephant 
> \src\elephant\serializer1.fasl
>
> 2. LispWorks 5 pro
>
> in file memutil.lisp
>
> This creates an error:
>
> (def-foreign-type array-or-pointer-char
>       #+allegro (:array :char)
>       #+(or cmu sbcl scl openmcl) (* :char))
>
> changed 2nd line to:
>       #+(or allegro lispworks) (:array :char)
> don't know whether it is the correct selection
>
> Then:
>
> Compilation aborted due to error between functions:
>   Package SB-KERNEL not found.
>
> when compiling this:
> ;; A non-back-compatible change was made in SBCL 8 moving to SBCL 9,
> ;; in that the function copy-from-system-area disappeared.
> ;; This code is an attempt to allow compilation under bothe SBCL 8  
> and SBCL 9.
> ;; Thanks to Juho Snellman for this idiom.
> (eval-when (:compile-toplevel)
>   (defun new-style-copy-p ()
>     (if (find-symbol "COPY-UB8-FROM-SYSTEM-AREA" "SB-KERNEL")
> 	'(:and)
> 	'(:or)))
>   )
>
> I then commented out all forms depending on
> ;#+#.(elephant-memutil::new-style-copy-p) and
> ;#-#.(elephant-memutil::new-style-copy-p)
>
> in file package lisp
>
> Compilation aborted due to error between functions:
>   Duplicated names in "ELEPHANT" defpackage: "ELE".
>
> I replaced
>  (:nicknames ele :ele)
> with
>  (:nicknames :ele)
>
> In file metaclasses.lisp
>
> **++++ Error in (DEFCLASS ELEPHANT:PERSISTENT):
>   Defining function :DBCN-SPC-PST visible from package KEYWORD.
>
> The accessor function stars with colon. I deleted the colon.
>
> In file classes.lisp
>
> Error
> Defining (METHOD ENSURE-CLASS-USING-CLASS :AROUND ((EQL NIL) T))  
> visible from packages COMMON-LISP, CLOS.
>
> No idea what to do.
>
>
> 3. I was able to use elephant 6.0 with ACL 7 trial
>
> wherein I used cygwin to produce the dll s, and I copied the dll,  
> with a newer date than the source, into the
> directory with binaries to prevent recompilation started with run- 
> shell-command from asdf, which does not seem
> to work on windows.
>
> This is the script for libmemutil.dll:
>
> gcc -mno-cygwin -mwindows -std=c99 -c libmemutil.c
> dlltool -z libmeutil.def --export-all-symbols -e exports.o -l  
> libmemutil.lib libmemutil.o
> gcc -shared -mno-cygwin -mwindows libmemutil.o exports.o -o  
> libmemutil.dll
>
> And this is the script for libsleepycat.dll:
>
> gcc -mno-cygwin -mwindows -c -Wall -std=c99 -L/c/DB/Berkeley\ DB\  
> 4.4.20/lib/ -I/c/DB/Berkeley\ DB\ 4.4.20/include/ libsleepycat.c
> dlltool -z libsleepycat.def --export-all-symbols -e exports.o -l  
> libsleepycat.lib libsleepycat.o
> gcc -shared -mno-cygwin -mwindows -L/c/DB/Berkeley\ DB\ 4.4.20/bin/  
> -llibdb44 libsleepycat.o exports.o -o libsleepycat.dll
>
> (I don't know how to use variables in shell scripts)
>
>
> 4. I will try again
>
> ______________________________________________________________________ 
> _
> Viren-Scan f?r Ihren PC! Jetzt f?r jeden. Sofort, online und  
> kostenlos.
> Gleich testen! http://www.pc-sicherheit.web.de/freescan/?mc=022222
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Wed Jan 24 01:09:23 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Tue, 23 Jan 2007 20:09:23 -0500
Subject: [elephant-devel] sbcl-64 tests
In-Reply-To: <1169599349.8265.229.camel@trinidad>
References: <1169599349.8265.229.camel@trinidad>
Message-ID: <F96607AB-7ED5-4B59-8BA3-A2CC72F58296@csail.mit.edu>

I was in the middle of doing some similar changes in memutil to  
support 64-bit (the files are still local on my machine).  Marco  
Baringer also did some work on 64-bit compliance.  Are you willing/ 
ready to forward your changes to me so we can have a discussion about  
the best way to update the CVS for 64-bit lisps?

I think I've seen the drop-instances problem before, but not for  
awhile.  I'll look into it as part of 0.6.1.  Is it intermittent or  
reproducible?

Thanks,
Ian

PS - On your multithreading observation.  Berkeley DB is definitely  
setup to use parallel processes.  Briefly, you link to the BDB  
library.  When you open a DB there's an on-disk environment that is  
initialized and shared among all processes.  This indexes into a  
shared memory region that is used to store locks, transactions,  
cached pages, logs, etc.  The BDB functions themselves handle  
blocking on locks and arbitrating between different processes.   
There's several ways to do this across machines as well (transaction  
server or distributed transactions) but I haven't looked into this much.

On Jan 23, 2007, at 7:42 PM, Henrik Hjelte wrote:

> I have started running the bdb backendtests on latest sbcl, amd64  
> linux.
> I also have got a beta snapshot of openmcl-linux-64 (2006-12-31)  
> running
> sometimes. I don't know how stable this beta is, but at least it's  
> good
> to run several lisps to find bugs.
>
> When running the backend tests for bdb on sbcl, the first failure was
> the FIXNUM test. It serializes/deserializes most-positive-fixnum.  
> Since
> most-positive-fixnum was larger than 32bit integers, I somehow  
> suspected
> 64 bit problems. Which was not the main problem. Instead it was  
> related
> to the type checking in the serializer. I have attached a solution  
> in a
> diff-file.
>
> (A parenthesis: My 64-bit adventures gave some code as a sideffect. I
> have added 64 bit integers to memutils.c and refactored/obscured it  
> with
> a macro that makes the reader and writer functions. I have changed all
> references to int to be explict int32 in memutils.lisp and
> bdb-controller. I have a half-finished version of bdb-controller with
> macros making the cursor movement methods. I'll keep it in the  
> closet if
> it's of any use later on. End of parenthesis)
>
> Now almost all backendtests run.
> 2 out of 115 total tests failed: INDEXING-WIPE-INDEX, INDEXING-TIMING.
> Index-timing fails with this error. This is something I have noted
> before with my recent experiments with elephant version 6. Suddently
> after deleting instances, elephant would sometimes say that the class
> was no longer indexed. I think it showed up with both the bdb and the
> clsql backend. Has no one else noticed this? Below is a backtrace, you
> can see the drop-instances call which I am suspecting.
>
> About multithreading/testing. I volunteer to add some kind of
> stress-test with bourdeaux threads later, but now the outstanding  
> bug is
> a blocker.
>
> Speaking of multithreading: since few Lisp:s use native threads, and
> writing thread-safe code can be difficult, in my opinion taking
> advantage of multi-core processors is most easily done with spreading
> processes. what about multiprocessing? I mean parallel Lisp instances,
> on the same or different computers, writing and reading to the same
> database? Is elephant supposed to work in that kind of environment. I
> suppose the clsql backend does, but what about bdb?
>
> That's all for now, it's in the middle of the night for me, got to  
> count
> some sheep.
> /Henrik Hjelte
>
>
> Class #<PERSISTENT-METACLASS STRESS-INDEX> is not an indexed class
>    [Condition of type SIMPLE-ERROR]
>
> Restarts:
>  0: [ABORT] Return to SLIME's top level.
>  1: [TERMINATE-THREAD] Terminate this thread (#<THREAD
> "worker" {1004E89801}>)
>
> Backtrace:
>   0: ((SB-PCL::FAST-METHOD FIND-CLASS-INDEX (PERSISTENT-METACLASS))
>       #<unused argument>
>       #<unused argument>
>       #<PERSISTENT-METACLASS STRESS-INDEX>
>       :SC
>       NIL
>       :ERRORP
>       T)
>   1: ((LAMBDA ()))
>   2: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
>        (DB-BDB::BDB-STORE-CONTROLLER #1="#<...>" . #1#))
>       #<unused argument>
>       #<unused argument>
>       #<DB-BDB::BDB-STORE-CONTROLLER {1004E9A2D1}>
>       #<CLOSURE (LAMBDA #) {10049CE119}>
>       :TRANSACTION
>       NIL
>       :ENVIRONMENT
>       NIL ..)
>   3: (DROP-INSTANCES
>       (#<STRESS-INDEX {10049CBB51}> #<STRESS-INDEX {10049CA331}>
>
>
>
>
>
>
> <diff-last-4-changes>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From ml13 at onlinehome.de  Wed Jan 24 10:24:11 2007
From: ml13 at onlinehome.de (Kilian Sprotte)
Date: Wed, 24 Jan 2007 11:24:11 +0100
Subject: [elephant-devel] osx 32bit testing
Message-ID: <3A585F34-CEB3-4397-8E1C-82B1472C910B@onlinehome.de>

Hi,

I really like the new installation process!! :)

First of all, I am attaching my-config.sexp that works for db44  
darwinports / macports :)

With sbcl I have all bdb tests passed!!!

For openmcl, I am attaching a diff, which allows it to pass some of  
the tests. Actually, isn't the type of the spec only cons now?

I am attaching the complete test dribble for openmcl as well. Sorry,  
I am in a hurry right now - I might find a better way to describe  
those type errors that appear in the the test results, or possibly  
send diffs....

Thanks a lot for all the work
on elephant!!

   Kilian

PS: One last issue, sorry. Wouldn't it be nice to specify  
only :depends-on (:ele-bdb) in a project that builds on ele? Right  
now, :elephant has to be loaded before :ele-bdb...


-------------- next part --------------
A non-text attachment was scrubbed...
Name: my-config.sexp
Type: application/octet-stream
Size: 493 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070124/911acf77/attachment.obj>
-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: ele.diff
Type: application/octet-stream
Size: 683 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070124/911acf77/attachment-0001.obj>
-------------- next part --------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: openmcl-test-result.zip
Type: application/zip
Size: 7130 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070124/911acf77/attachment.zip>

From ml13 at onlinehome.de  Wed Jan 24 10:43:41 2007
From: ml13 at onlinehome.de (Kilian Sprotte)
Date: Wed, 24 Jan 2007 11:43:41 +0100
Subject: [elephant-devel] osx 32bit testing
In-Reply-To: <3A585F34-CEB3-4397-8E1C-82B1472C910B@onlinehome.de>
References: <3A585F34-CEB3-4397-8E1C-82B1472C910B@onlinehome.de>
Message-ID: <186D2C22-F6D5-47D3-84BF-26A1CDA74BC3@onlinehome.de>


Am 24.01.2007 um 11:24 schrieb Kilian Sprotte:

> I am attaching the complete test dribble for openmcl as well.  
> Sorry, I am in a hurry right now - I might find a better way to  
> describe those type errors that appear in the the test results, or  
> possibly send diffs....

Actually, the first type error that appears has this backtrace:
(probably, we just need unicode and I should try the beta openmcl 1.1,
since I am on 1.0 right now ?)

value 116 is not of the expected type CCL:MACPTR.
    [Condition of type TYPE-ERROR]

Restarts:
0: [ABORT] Return to SLIME's top level.
1: [ABORT-BREAK] Reset this process
2: [ABORT] Kill this process

Backtrace:
   0: (ELEPHANT-SERIALIZER2::SERIALIZE-TO-UTF8 "this is a test" #S 
(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER #<A Mac Pointer #x180AA00>
                                        :SIZE 5
                                        :POSITION 0
                                        :LENGTH 1280))
   1: (ELEPHANT-SERIALIZER2::SERIALIZE "this is a test" #<DB-BDB::BDB- 
STORE-CONTROLLER #x89F170E> '(NULL (PROGN (IN-OUT-EQUAL (COERCE "this  
is a test" 'BASE-STRING)))))
       Locals:
         ELEPHANT-SERIALIZER2::FROB = "this is a test"
         ELEPHANT-SERIALIZER2::SC = #<DB-BDB::BDB-STORE-CONTROLLER  
#x89F170E>
         ELEPHANT-SERIALIZER2::*LISP-OBJ-ID* = #<BOGUS object @  
#x14EF8DE>
         ELEPHANT-SERIALIZER2::*CIRCULARITY-HASH* = #<HASH- 
TABLE :TEST EQ size 0/50 #x89F6986>
   2: (IN-OUT-EQUAL "this is a test")
       Locals:
         VAR = "this is a test"
         OUT-BUF = #S(ELEPHANT-MEMUTIL:BUFFER-STREAM :BUFFER #<A Mac  
Pointer #x180AA00>
		                                    :SIZE 5
		                                    :POSITION 0
		                                    :LENGTH 1280)
       Catch-tags:
         NIL
   3: (CCL::CALL-CHECK-REGS 'IN-OUT-EQUAL)
       Locals:
         CCL::FN = IN-OUT-EQUAL
         CCL::ARGS = ("this is a test")
         CCL::OLD-REGS = ((NULL #1=(PROGN (IN-OUT-EQUAL #))) NIL NIL  
NULL NIL #1# 0
			 MACROEXPAND-1)
   4: (CCL::CHEAP-EVAL-IN-ENVIRONMENT '(NOT (NULL (PROGN (IN-OUT- 
EQUAL #)))) 'NIL)
   5: (CCL::CHEAP-EVAL-IN-ENVIRONMENT '(VALUES (IS-NOT-NULL (IN-OUT- 
EQUAL (MAKE-STRING
                                           0
                                           :ELEMENT-TYPE
                                           'BASE-CHAR)))
               (IS-NOT-NULL (IN-OUT-EQUAL (COERCE
                                           "this is a test"
                                           'BASE-STRING)))
               (IS-NOT-NULL (IN-OUT-EQUAL (MAKE-STRING
                                           400
                                           :INITIAL-ELEMENT ..))))  
'((IS-NOT-NULL (IN-OUT-EQUAL (MAKE-STRING ..)
   6: (CCL::CHEAP-EVAL-IN-ENVIRONMENT #S(REGRESSION-TEST::ENTRY :PEND  
T :NAME BASE-STRINGS :PROPS NIL
                                :FORM (ARE-NOT-NULL
                                       (IN-OUT-EQUAL
                                        (MAKE-STRING
                                         0
                                         :ELEMENT-TYPE
                                         'BASE-CHAR))
                                       (IN-OUT-EQUAL
                                        (COERCE
                                         "this is a test" ..))))) '(#S 
(REGRESSION-TEST::ENTRY :PEND T :NAME STRINGS :PROPS NIL ..)
   7: (REGRESSION-TEST::%DO #S(REGRESSION-TEST::ENTRY :PEND T :NAME  
BASE-STRINGS :PROPS NIL
                                :FORM (ARE-NOT-NULL
                                       (IN-OUT-EQUAL
                                        (MAKE-STRING
                                         0
                                         :ELEMENT-TYPE
                                         'BASE-CHAR))
                                       (IN-OUT-EQUAL
                                        (COERCE
                                         "this is a test" ..))))))
   8: (REGRESSION-TEST::DO-ENTRY #S(REGRESSION-TEST::ENTRY :PEND  
T :NAME BASE-STRINGS :PROPS NIL
                                :FORM (ARE-NOT-NULL
                                       (IN-OUT-EQUAL
                                        (MAKE-STRING
                                         0
                                         :ELEMENT-TYPE
                                         'BASE-CHAR))
                                       (IN-OUT-EQUAL
                                        (COERCE
                                         "this is a test" ..)))))  
#<SWANK-BACKEND::SLIME-OUTPUT-STREAM #x863CD0E>)
   9: (REGRESSION-TEST::DO-ENTRIES* '(:BDB "/private/tmp/elephant/ 
tests/testdb/"))
10: (DO-BACKEND-TESTS)
11: (CCL::CALL-CHECK-REGS 'DO-BACKEND-TESTS)
12: (SWANK::EVAL-REGION "(do-backend-tests)
      " 'T)
13: (#<Anonymous Function #x860B06E> "(do-backend-tests)
      ")
14: (SWANK::CALL-WITH-BUFFER-SYNTAX #<CCL:COMPILED-LEXICAL-CLOSURE  
#x89F17BE>)
15: (SWANK:LISTENER-EVAL "(do-backend-tests)
      ")
...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070124/c87ef2f5/attachment.html>

From henrik at evahjelte.com  Wed Jan 24 11:40:38 2007
From: henrik at evahjelte.com (Henrik Hjelte)
Date: Wed, 24 Jan 2007 12:40:38 +0100
Subject: [elephant-devel] sbcl-64 tests
In-Reply-To: <F96607AB-7ED5-4B59-8BA3-A2CC72F58296@csail.mit.edu>
References: <1169599349.8265.229.camel@trinidad>
	<F96607AB-7ED5-4B59-8BA3-A2CC72F58296@csail.mit.edu>
Message-ID: <1169638838.8265.243.camel@trinidad>

On Tue, 2007-01-23 at 20:09 -0500, Ian Eslick wrote:
> I was in the middle of doing some similar changes in memutil to  
> support 64-bit (the files are still local on my machine).  Marco  
> Baringer also did some work on 64-bit compliance.  Are you willing/ 
> ready to forward your changes to me so we can have a discussion about  
> the best way to update the CVS for 64-bit lisps?
I've mailed them to you.

> 
> I think I've seen the drop-instances problem before, but not for  
> awhile.  I'll look into it as part of 0.6.1.  Is it intermittent or  
> reproducible?
It is reproducible for me with sbcl/amd64/linux. At least
INDEXING-TIMING fails every time. If you can't reproduce it I can set up
some remote access to my machine.

Best wishes,
Henrik




From lists at infoway.net  Wed Jan 24 16:22:43 2007
From: lists at infoway.net (lists at infoway.net)
Date: Wed, 24 Jan 2007 11:22:43 -0500 (EST)
Subject: [elephant-devel] 0.6.1 / OpenMCL / BDB
Message-ID: <50607.192.168.1.35.1169655763.webmail@192.168.1.35>

I have been trying to make 0.6.1 work under OpenMCL Version 1.1-pre-061231 (DarwinX8664). However, as I mentioned before, I have been having compilation problems.

They seem to be mainly related to libmemutil.

Out of the box attempt, I got the following error:

; $ /usr/bin/gcc -shared -Wall -fPIC -O3 -o /Users/dev/lisp/elephant/src/memutil/libmemutil.so /Users/dev/lisp/elephant/src/memutil/libmemutil.c -lm 
i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
/usr/bin/ld: Undefined symbols:
_main
collect2: ld returned 1 exit status

Upon inspection of elephant.asd, I made the following change:

(defmethod compiler-options ((compiler (eql :gcc)) (c elephant-c-source) &key input-file output-file)
  "Default compile and link options to create a library; no -L or -I options included; math lib as default"
  (unless (and input-file output-file)
    (error "Must specify both input and output files"))
  (list
   #-(or openmcl darwin macosx) "-shared"
   #+(or openmcl darwin macosx) "-bundle"
   "-Wall"
   "-fPIC"
   "-O3"
   "-o" output-file
   input-file
   "-lm"))

What I basically did is I added "openmcl" to the conditionalization #- and #+. After making this change, it successfully builds the libmemutil.so file. However, while the file actually exists, it fails to load with the following message:

Error opening shared library "/Users/dev/lisp/elephant/src/memutil/libmemutil.so": dlopen(/Users/dev/lisp/elephant/src/memutil/libmemutil.so, 10): no suitable image found.  Did find:
	/Users/dev/lisp/elephant/src/memutil/libmemutil.so: mach-o, but wrong architecture
   [Condition of type SIMPLE-ERROR]

FYI, my *FEATURES* is:

(:KMR-NORMAL-DSDC :KMR-NORMAL-CESD :KMR-MOP :ASDF :PRIMARY-CLASSES :COMMON-LISP :OPENMCL :CLOZURE :ANSI-CL :UNIX :OPENMCL-NATIVE-THREADS :OPENMCL-PARTIAL-MOP :MCL-COMMON-MOP-SUBSET :OPENMCL-MOP-2 :OPENMCL-PRIVATE-HASH-TABLES :OPENMCL-HASH-CONSING :X86-64 :X86-64 :X86-TARGET :X86-HOST :X8664-TARGET :X8664-HOST :DARWIN-HOST :DARWIN-TARGET :DARWINX86-TARGET :DARWINX8664-TARGET :DARWINX8664-HOST :POWEROPEN-TARGET :64-BIT-TARGET :64-BIT-HOST :LITTLE-ENDIAN-TARGET :LITTLE-ENDIAN-HOST :MCL)

I don't really know why it cannot load the shared library. I don't know if the problem is basically a compat issue with this CVS version of OpenMCL in this architecture and/or UFFI. However, I thought I would share this just in case anyone has information that could speed this process or any suggestions of what I may try next.

Thanks,
Daniel



From eslick at csail.mit.edu  Wed Jan 24 16:39:40 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Wed, 24 Jan 2007 11:39:40 -0500
Subject: [elephant-devel] 0.6.1 / OpenMCL / BDB
In-Reply-To: <50607.192.168.1.35.1169655763.webmail@192.168.1.35>
References: <50607.192.168.1.35.1169655763.webmail@192.168.1.35>
Message-ID: <14D2D58C-F167-4CEB-95FC-8FA2AD7070DE@csail.mit.edu>

This is a problem with assumptions about *features*

On Darwin we should be loading .dylib which is generated by the - 
bundle option to gcc/ld

UFFI looks for tags in *features* as follows to determine the lib  
type to load:

(defun foreign-library-types ()
   "Returns list of string naming possible library types for  
platform, sorted by preference"
   #+(or win32 mswindows) '("dll" "lib")
   #+(or macosx darwin ccl-5.0) '("dylib" "bundle")
   #-(or win32 mswindows macosx darwin ccl-5.0) '("so" "a" "o")
)

Our compile script in elephant.asd relies on darwin/macosx as the  
indicator as to what kind of library to produce (-shared vs. - 
bundle).  I have no idea why OpenMCL changed the *features* list!   
Very annoying.

I notice that :darwin-host is on the list to differentiate  
from :darwin-target, presumably to allow cross-compilation.

I think the right solution is to push :macosx onto the *features*  
list for now so everything just works.  Do this before you load  
elephant.  Perhaps we can find out from the OpenMCL folks what the  
right *feature* is to rely on before we modify our script and ask the  
UFFI maintainer to add new tags to update foreign-library-types to  
accommodate OpenMCL 1.1

Ian

On Jan 24, 2007, at 11:22 AM, lists at infoway.net wrote:

> I have been trying to make 0.6.1 work under OpenMCL Version 1.1- 
> pre-061231 (DarwinX8664). However, as I mentioned before, I have  
> been having compilation problems.
>
> They seem to be mainly related to libmemutil.
>
> Out of the box attempt, I got the following error:
>
> ; $ /usr/bin/gcc -shared -Wall -fPIC -O3 -o /Users/dev/lisp/ 
> elephant/src/memutil/libmemutil.so /Users/dev/lisp/elephant/src/ 
> memutil/libmemutil.c -lm
> i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
> /usr/bin/ld: Undefined symbols:
> _main
> collect2: ld returned 1 exit status
>
> Upon inspection of elephant.asd, I made the following change:
>
> (defmethod compiler-options ((compiler (eql :gcc)) (c elephant-c- 
> source) &key input-file output-file)
>   "Default compile and link options to create a library; no -L or - 
> I options included; math lib as default"
>   (unless (and input-file output-file)
>     (error "Must specify both input and output files"))
>   (list
>    #-(or openmcl darwin macosx) "-shared"
>    #+(or openmcl darwin macosx) "-bundle"
>    "-Wall"
>    "-fPIC"
>    "-O3"
>    "-o" output-file
>    input-file
>    "-lm"))
>
> What I basically did is I added "openmcl" to the conditionalization  
> #- and #+. After making this change, it successfully builds the  
> libmemutil.so file. However, while the file actually exists, it  
> fails to load with the following message:
>
> Error opening shared library "/Users/dev/lisp/elephant/src/memutil/ 
> libmemutil.so": dlopen(/Users/dev/lisp/elephant/src/memutil/ 
> libmemutil.so, 10): no suitable image found.  Did find:
> 	/Users/dev/lisp/elephant/src/memutil/libmemutil.so: mach-o, but  
> wrong architecture
>    [Condition of type SIMPLE-ERROR]
>
> FYI, my *FEATURES* is:
>
> (:KMR-NORMAL-DSDC :KMR-NORMAL-CESD :KMR-MOP :ASDF :PRIMARY- 
> CLASSES :COMMON-LISP :OPENMCL :CLOZURE :ANSI-CL :UNIX :OPENMCL- 
> NATIVE-THREADS :OPENMCL-PARTIAL-MOP :MCL-COMMON-MOP-SUBSET :OPENMCL- 
> MOP-2 :OPENMCL-PRIVATE-HASH-TABLES :OPENMCL-HASH- 
> CONSING :X86-64 :X86-64 :X86-TARGET :X86-HOST :X8664-TARGET :X8664- 
> HOST :DARWIN-HOST :DARWIN-TARGET :DARWINX86-TARGET :DARWINX8664- 
> TARGET :DARWINX8664-HOST :POWEROPEN-TARGET :64-BIT-TARGET :64-BIT- 
> HOST :LITTLE-ENDIAN-TARGET :LITTLE-ENDIAN-HOST :MCL)
>
> I don't really know why it cannot load the shared library. I don't  
> know if the problem is basically a compat issue with this CVS  
> version of OpenMCL in this architecture and/or UFFI. However, I  
> thought I would share this just in case anyone has information that  
> could speed this process or any suggestions of what I may try next.
>
> Thanks,
> Daniel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From lists at infoway.net  Wed Jan 24 17:51:50 2007
From: lists at infoway.net (lists at infoway.net)
Date: Wed, 24 Jan 2007 12:51:50 -0500 (EST)
Subject: [elephant-devel] 0.6.1 / OpenMCL / BDB
Message-ID: <39982.192.168.1.35.1169661110.webmail@192.168.1.35>

I did (push :macosx *features*) as well as (push :darwin *features*) and in both instances, it failed to load the .so library. I guess somehow, UFFI still doesn't recognize it to use .dylib (and .dylib was not generated either).

Very strange. Is there another way of "pushing" stuff into *features* that I'm not aware of?

Thanks,
Daniel


On Wed, January 24, 2007 11:39 am, Ian Eslick <eslick at csail.mit.edu> said:

> This is a problem with assumptions about *features*
> 
> On Darwin we should be loading .dylib which is generated by the -
> bundle option to gcc/ld
> 
> UFFI looks for tags in *features* as follows to determine the lib
> type to load:
> 
> (defun foreign-library-types ()
>    "Returns list of string naming possible library types for
> platform, sorted by preference"
>    #+(or win32 mswindows) '("dll" "lib")
>    #+(or macosx darwin ccl-5.0) '("dylib" "bundle")
>    #-(or win32 mswindows macosx darwin ccl-5.0) '("so" "a" "o")
> )
> 
> Our compile script in elephant.asd relies on darwin/macosx as the
> indicator as to what kind of library to produce (-shared vs. -
> bundle).  I have no idea why OpenMCL changed the *features* list!
> Very annoying.
> 
> I notice that :darwin-host is on the list to differentiate
> from :darwin-target, presumably to allow cross-compilation.
> 
> I think the right solution is to push :macosx onto the *features*
> list for now so everything just works.  Do this before you load
> elephant.  Perhaps we can find out from the OpenMCL folks what the
> right *feature* is to rely on before we modify our script and ask the
> UFFI maintainer to add new tags to update foreign-library-types to
> accommodate OpenMCL 1.1
> 
> Ian
> 
> On Jan 24, 2007, at 11:22 AM, lists at infoway.net wrote:
> 
>> I have been trying to make 0.6.1 work under OpenMCL Version 1.1-
>> pre-061231 (DarwinX8664). However, as I mentioned before, I have
>> been having compilation problems.
>>
>> They seem to be mainly related to libmemutil.
>>
>> Out of the box attempt, I got the following error:
>>
>> ; $ /usr/bin/gcc -shared -Wall -fPIC -O3 -o /Users/dev/lisp/
>> elephant/src/memutil/libmemutil.so /Users/dev/lisp/elephant/src/
>> memutil/libmemutil.c -lm
>> i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
>> /usr/bin/ld: Undefined symbols:
>> _main
>> collect2: ld returned 1 exit status
>>
>> Upon inspection of elephant.asd, I made the following change:
>>
>> (defmethod compiler-options ((compiler (eql :gcc)) (c elephant-c-
>> source) &key input-file output-file)
>>   "Default compile and link options to create a library; no -L or -
>> I options included; math lib as default"
>>   (unless (and input-file output-file)
>>     (error "Must specify both input and output files"))
>>   (list
>>    #-(or openmcl darwin macosx) "-shared"
>>    #+(or openmcl darwin macosx) "-bundle"
>>    "-Wall"
>>    "-fPIC"
>>    "-O3"
>>    "-o" output-file
>>    input-file
>>    "-lm"))
>>
>> What I basically did is I added "openmcl" to the conditionalization
>> #- and #+. After making this change, it successfully builds the
>> libmemutil.so file. However, while the file actually exists, it
>> fails to load with the following message:
>>
>> Error opening shared library "/Users/dev/lisp/elephant/src/memutil/
>> libmemutil.so": dlopen(/Users/dev/lisp/elephant/src/memutil/
>> libmemutil.so, 10): no suitable image found.  Did find:
>> 	/Users/dev/lisp/elephant/src/memutil/libmemutil.so: mach-o, but
>> wrong architecture
>>    [Condition of type SIMPLE-ERROR]
>>
>> FYI, my *FEATURES* is:
>>
>> (:KMR-NORMAL-DSDC :KMR-NORMAL-CESD :KMR-MOP :ASDF :PRIMARY-
>> CLASSES :COMMON-LISP :OPENMCL :CLOZURE :ANSI-CL :UNIX :OPENMCL-
>> NATIVE-THREADS :OPENMCL-PARTIAL-MOP :MCL-COMMON-MOP-SUBSET :OPENMCL-
>> MOP-2 :OPENMCL-PRIVATE-HASH-TABLES :OPENMCL-HASH-
>> CONSING :X86-64 :X86-64 :X86-TARGET :X86-HOST :X8664-TARGET :X8664-
>> HOST :DARWIN-HOST :DARWIN-TARGET :DARWINX86-TARGET :DARWINX8664-
>> TARGET :DARWINX8664-HOST :POWEROPEN-TARGET :64-BIT-TARGET :64-BIT-
>> HOST :LITTLE-ENDIAN-TARGET :LITTLE-ENDIAN-HOST :MCL)
>>
>> I don't really know why it cannot load the shared library. I don't
>> know if the problem is basically a compat issue with this CVS
>> version of OpenMCL in this architecture and/or UFFI. However, I
>> thought I would share this just in case anyone has information that
>> could speed this process or any suggestions of what I may try next.
>>
>> Thanks,
>> Daniel
>>
>> _______________________________________________
>> elephant-devel site list
>> elephant-devel at common-lisp.net
>> http://common-lisp.net/mailman/listinfo/elephant-devel
> 
> 



From nowhere.man at levallois.eu.org  Thu Jan 25 23:40:33 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Fri, 26 Jan 2007 00:40:33 +0100
Subject: [elephant-devel] Bug in string serialisation?
Message-ID: <20070125234033.GB18824@bateleur.arcanes.fr.eu.org>

I'm working on a web application that uses 0.6.0, and I may have hit a
bug in Elephant.

I have a fairly reproducible bug, when storing a string. I sometimes
have to decode a badly-read string. E.g.:

- I have "Id????al"
- I want "Id??al"

For this I use a function that if the ?? character is found, decode the
string from UTF-8.

(decode-string-if-needed "Id??al") => "Id??al"
(decode-string-if-needed "Id????al") => "Id??al"

The problem is when I store the result in a slot of a persistent class.
I tried to store it manually quite some times, I never had any problem:

(setf (product-description p) "Id??al")
(product-description p) => "Id??al"

As illogic as it seems, if the slot is "Id????al", the following:

(setf (product-description p) (decode-string-if-needed (product-description p)))

doesn't have the same result. If I retrieve the string from the slot,
usually swank deconnects because it encountered strange characters.

(map 'vector #'char-code "Id??al") => #(73 100 233 97 108)
(map 'vector #'char-code (product-description p)) => #(39 24 8483047 0 11)


I wrote the following test macro:

(defmacro test-conversion (location)
 `(let* ((bad ,location)
         (good (decode-string-if-needed bad))
    (setf ,location good)
    (let ((stored ,location))
      (mapcar (lambda (string) (map 'vector #'char-code string)) (list bad good stored))))))

And I reliably got the following:

(setf (product-description p) "Id????al")
(test-conversion (product-description p))
=> (#(73 100 195 169 97 108)
    #(73 100 233 97 108)
    #(39 24 15187975 0 11))

Does someone understand what could be going on?

Strangely,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070126/f4ce5f12/attachment.sig>

From eslick at csail.mit.edu  Fri Jan 26 00:59:10 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Thu, 25 Jan 2007 19:59:10 -0500
Subject: [elephant-devel] Bug in string serialisation?
In-Reply-To: <20070125234033.GB18824@bateleur.arcanes.fr.eu.org>
References: <20070125234033.GB18824@bateleur.arcanes.fr.eu.org>
Message-ID: <F8DC613E-EF97-492B-8A66-D5DB9478C5FE@csail.mit.edu>

We need a good set of unicode tests!  I redesigned unicode support in  
0.6.1 so I'm hoping these issues will go away, but I'd like to  
understand it.

What lisp are you using and what character coding, etc?

Ian

On Jan 25, 2007, at 6:40 PM, Pierre THIERRY wrote:

> I'm working on a web application that uses 0.6.0, and I may have hit a
> bug in Elephant.
>
> I have a fairly reproducible bug, when storing a string. I sometimes
> have to decode a badly-read string. E.g.:
>
> - I have "Id????al"
> - I want "Id??al"
>
> For this I use a function that if the ?? character is found, decode  
> the
> string from UTF-8.
>
> (decode-string-if-needed "Id??al") => "Id??al"
> (decode-string-if-needed "Id????al") => "Id??al"
>
> The problem is when I store the result in a slot of a persistent  
> class.
> I tried to store it manually quite some times, I never had any  
> problem:
>
> (setf (product-description p) "Id??al")
> (product-description p) => "Id??al"
>
> As illogic as it seems, if the slot is "Id????al", the following:
>
> (setf (product-description p) (decode-string-if-needed (product- 
> description p)))
>
> doesn't have the same result. If I retrieve the string from the slot,
> usually swank deconnects because it encountered strange characters.
>
> (map 'vector #'char-code "Id??al") => #(73 100 233 97 108)
> (map 'vector #'char-code (product-description p)) => #(39 24  
> 8483047 0 11)
>
>
> I wrote the following test macro:
>
> (defmacro test-conversion (location)
>  `(let* ((bad ,location)
>          (good (decode-string-if-needed bad))
>     (setf ,location good)
>     (let ((stored ,location))
>       (mapcar (lambda (string) (map 'vector #'char-code string))  
> (list bad good stored))))))
>
> And I reliably got the following:
>
> (setf (product-description p) "Id????al")
> (test-conversion (product-description p))
> => (#(73 100 195 169 97 108)
>     #(73 100 233 97 108)
>     #(39 24 15187975 0 11))
>
> Does someone understand what could be going on?
>
> Strangely,
> Pierre
> -- 
> nowhere.man at levallois.eu.org
> OpenPGP 0xD9D50D8A
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: PGP.sig
Type: application/pgp-signature
Size: 186 bytes
Desc: This is a digitally signed message part
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070125/e5bd75a1/attachment.sig>

From nowhere.man at levallois.eu.org  Fri Jan 26 07:04:43 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Fri, 26 Jan 2007 08:04:43 +0100
Subject: [elephant-devel] Bug in string serialisation?
In-Reply-To: <20070125234033.GB18824@bateleur.arcanes.fr.eu.org>
References: <20070125234033.GB18824@bateleur.arcanes.fr.eu.org>
Message-ID: <20070126070443.GC18824@bateleur.arcanes.fr.eu.org>

Scribit Pierre THIERRY dies 26/01/2007 hora 00:40:
> - I have "Id????al"
> - I want "Id??al"

Oh dear. My mail was sent in UTF-8 as Latin-1... I *hate* encoding
issues!

Quickly,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070126/1915d9ff/attachment.sig>

From nowhere.man at levallois.eu.org  Fri Jan 26 07:08:31 2007
From: nowhere.man at levallois.eu.org (Pierre THIERRY)
Date: Fri, 26 Jan 2007 08:08:31 +0100
Subject: [elephant-devel] Bug in string serialisation?
In-Reply-To: <F8DC613E-EF97-492B-8A66-D5DB9478C5FE@csail.mit.edu>
References: <20070125234033.GB18824@bateleur.arcanes.fr.eu.org>
	<F8DC613E-EF97-492B-8A66-D5DB9478C5FE@csail.mit.edu>
Message-ID: <20070126070831.GD18824@bateleur.arcanes.fr.eu.org>

Scribit Ian Eslick dies 25/01/2007 hora 19:59:
> We need a good set of unicode tests!  I redesigned unicode support in
> 0.6.1 so I'm hoping these issues will go away, but I'd like to
> understand it.

I'll make some tests of my current application under 0.6.1 to see.

> What lisp are you using and what character coding, etc?

I'm using SBCL 1.0.0 and all content read from and written to the
outside is according UTF-8. IIUC, SBCL's character are independent of
the encoding used, but in the Unicode table (for their character code).

Quickly,
Pierre
-- 
nowhere.man at levallois.eu.org
OpenPGP 0xD9D50D8A
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20070126/9e6183fc/attachment.sig>

From eslick at csail.mit.edu  Fri Jan 26 10:03:16 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Fri, 26 Jan 2007 05:03:16 -0500
Subject: [elephant-devel] 0.6.1 / OpenMCL / BDB
In-Reply-To: <50607.192.168.1.35.1169655763.webmail@192.168.1.35>
References: <50607.192.168.1.35.1169655763.webmail@192.168.1.35>
Message-ID: <40C6BE1B-609F-46D6-8DFA-E23494C65210@csail.mit.edu>

I finally figured out what was going on.  I hadn't realized that 1.1  
was only a x86_64 distribution (will only work on the new Core 2 Duo  
Intel chips or old PPC chips).

So there's a nice set of problems:

OpenMCL needs to link to 64-bit libraries, so we need to build 64-bit  
versions of libmemutil and libberkeley-db.

So in elephant.asd we need to add a line to the gcc compiler options:
   (list
    #-(or darwin macosx darwin-host) "-shared"
    #+(or darwin macosx darwin-host) "-bundle"
    #+(or x8664-target) "-arch x86_64"
    "-Wall"
    "-fPIC"
    "-O3"
    "-o" output-file
    input-file
    "-lm"))

So that's great and I can load libmemutil without a problem.   
However, when it comes time to load ele-bdb, you also need a 64-bit  
version of berkeley db for the -ldb option, so you're going to need  
to build or download a 64-bit version of that.

I don't have time to do that just now, but if you want to give that a  
try and let me know how it works that would be helpful!

Ian


On Jan 24, 2007, at 11:22 AM, lists at infoway.net wrote:

> I have been trying to make 0.6.1 work under OpenMCL Version 1.1- 
> pre-061231 (DarwinX8664). However, as I mentioned before, I have  
> been having compilation problems.
>
> They seem to be mainly related to libmemutil.
>
> Out of the box attempt, I got the following error:
>
> ; $ /usr/bin/gcc -shared -Wall -fPIC -O3 -o /Users/dev/lisp/ 
> elephant/src/memutil/libmemutil.so /Users/dev/lisp/elephant/src/ 
> memutil/libmemutil.c -lm
> i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
> /usr/bin/ld: Undefined symbols:
> _main
> collect2: ld returned 1 exit status
>
> Upon inspection of elephant.asd, I made the following change:
>
> (defmethod compiler-options ((compiler (eql :gcc)) (c elephant-c- 
> source) &key input-file output-file)
>   "Default compile and link options to create a library; no -L or - 
> I options included; math lib as default"
>   (unless (and input-file output-file)
>     (error "Must specify both input and output files"))
>   (list
>    #-(or openmcl darwin macosx) "-shared"
>    #+(or openmcl darwin macosx) "-bundle"
>    "-Wall"
>    "-fPIC"
>    "-O3"
>    "-o" output-file
>    input-file
>    "-lm"))
>
> What I basically did is I added "openmcl" to the conditionalization  
> #- and #+. After making this change, it successfully builds the  
> libmemutil.so file. However, while the file actually exists, it  
> fails to load with the following message:
>
> Error opening shared library "/Users/dev/lisp/elephant/src/memutil/ 
> libmemutil.so": dlopen(/Users/dev/lisp/elephant/src/memutil/ 
> libmemutil.so, 10): no suitable image found.  Did find:
> 	/Users/dev/lisp/elephant/src/memutil/libmemutil.so: mach-o, but  
> wrong architecture
>    [Condition of type SIMPLE-ERROR]
>
> FYI, my *FEATURES* is:
>
> (:KMR-NORMAL-DSDC :KMR-NORMAL-CESD :KMR-MOP :ASDF :PRIMARY- 
> CLASSES :COMMON-LISP :OPENMCL :CLOZURE :ANSI-CL :UNIX :OPENMCL- 
> NATIVE-THREADS :OPENMCL-PARTIAL-MOP :MCL-COMMON-MOP-SUBSET :OPENMCL- 
> MOP-2 :OPENMCL-PRIVATE-HASH-TABLES :OPENMCL-HASH- 
> CONSING :X86-64 :X86-64 :X86-TARGET :X86-HOST :X8664-TARGET :X8664- 
> HOST :DARWIN-HOST :DARWIN-TARGET :DARWINX86-TARGET :DARWINX8664- 
> TARGET :DARWINX8664-HOST :POWEROPEN-TARGET :64-BIT-TARGET :64-BIT- 
> HOST :LITTLE-ENDIAN-TARGET :LITTLE-ENDIAN-HOST :MCL)
>
> I don't really know why it cannot load the shared library. I don't  
> know if the problem is basically a compat issue with this CVS  
> version of OpenMCL in this architecture and/or UFFI. However, I  
> thought I would share this just in case anyone has information that  
> could speed this process or any suggestions of what I may try next.
>
> Thanks,
> Daniel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel



From eslick at csail.mit.edu  Wed Jan 31 21:51:39 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Wed, 31 Jan 2007 16:51:39 -0500
Subject: [elephant-devel] BDB users take note
Message-ID: <11E6B4AF-38DD-4230-9F78-95D4228B967C@csail.mit.edu>

The current CVS HEAD has been updated to require BDB 4.5 instead of  
BDB 4.4.  Elephant will continue to track BDB releases and only  
support the latest one.  The next time there is a major version  
update, we may consider supporting more than one but to do so  
requires more work than the maintainers currently are willing to put in.

Thank you,
Ian


From eslick at csail.mit.edu  Wed Jan 31 22:32:50 2007
From: eslick at csail.mit.edu (Ian Eslick)
Date: Wed, 31 Jan 2007 17:32:50 -0500
Subject: [elephant-devel] 0.6.1 / OpenMCL / BDB
In-Reply-To: <50607.192.168.1.35.1169655763.webmail@192.168.1.35>
References: <50607.192.168.1.35.1169655763.webmail@192.168.1.35>
Message-ID: <5CD02999-5EEB-45AF-B5C9-CC397AF34E7C@csail.mit.edu>

I can get OpenMCL 1.1 to compile build and launch, but BDB cannot  
open a newly created DB in open-store-controller.  I'm wondering if  
this is due to UFFI's handling of types such that we're getting 32- 
bit instead of 64-bit pointers out of the underlying C API (int vs.  
long)?

The steps to getting this far are:
OpenMCL 1.1 for x86-64
Configure and compile BDB 4.5 with -arch x86_64
    (something like: env LDFLAGS="-arch x86_64" CFLAGS="-arch x86_64"  
configure from build_unix)
Latest Elephant 0.6.1 from CVS (as of 1/31/07)

Daniel, want to take another crack at this?

Thanks,
Ian

On Jan 24, 2007, at 11:22 AM, lists at infoway.net wrote:

> I have been trying to make 0.6.1 work under OpenMCL Version 1.1- 
> pre-061231 (DarwinX8664). However, as I mentioned before, I have  
> been having compilation problems.
>
> They seem to be mainly related to libmemutil.
>
> Out of the box attempt, I got the following error:
>
> ; $ /usr/bin/gcc -shared -Wall -fPIC -O3 -o /Users/dev/lisp/ 
> elephant/src/memutil/libmemutil.so /Users/dev/lisp/elephant/src/ 
> memutil/libmemutil.c -lm
> i686-apple-darwin8-gcc-4.0.1: unrecognized option '-shared'
> /usr/bin/ld: Undefined symbols:
> _main
> collect2: ld returned 1 exit status
>
> Upon inspection of elephant.asd, I made the following change:
>
> (defmethod compiler-options ((compiler (eql :gcc)) (c elephant-c- 
> source) &key input-file output-file)
>   "Default compile and link options to create a library; no -L or - 
> I options included; math lib as default"
>   (unless (and input-file output-file)
>     (error "Must specify both input and output files"))
>   (list
>    #-(or openmcl darwin macosx) "-shared"
>    #+(or openmcl darwin macosx) "-bundle"
>    "-Wall"
>    "-fPIC"
>    "-O3"
>    "-o" output-file
>    input-file
>    "-lm"))
>
> What I basically did is I added "openmcl" to the conditionalization  
> #- and #+. After making this change, it successfully builds the  
> libmemutil.so file. However, while the file actually exists, it  
> fails to load with the following message:
>
> Error opening shared library "/Users/dev/lisp/elephant/src/memutil/ 
> libmemutil.so": dlopen(/Users/dev/lisp/elephant/src/memutil/ 
> libmemutil.so, 10): no suitable image found.  Did find:
> 	/Users/dev/lisp/elephant/src/memutil/libmemutil.so: mach-o, but  
> wrong architecture
>    [Condition of type SIMPLE-ERROR]
>
> FYI, my *FEATURES* is:
>
> (:KMR-NORMAL-DSDC :KMR-NORMAL-CESD :KMR-MOP :ASDF :PRIMARY- 
> CLASSES :COMMON-LISP :OPENMCL :CLOZURE :ANSI-CL :UNIX :OPENMCL- 
> NATIVE-THREADS :OPENMCL-PARTIAL-MOP :MCL-COMMON-MOP-SUBSET :OPENMCL- 
> MOP-2 :OPENMCL-PRIVATE-HASH-TABLES :OPENMCL-HASH- 
> CONSING :X86-64 :X86-64 :X86-TARGET :X86-HOST :X8664-TARGET :X8664- 
> HOST :DARWIN-HOST :DARWIN-TARGET :DARWINX86-TARGET :DARWINX8664- 
> TARGET :DARWINX8664-HOST :POWEROPEN-TARGET :64-BIT-TARGET :64-BIT- 
> HOST :LITTLE-ENDIAN-TARGET :LITTLE-ENDIAN-HOST :MCL)
>
> I don't really know why it cannot load the shared library. I don't  
> know if the problem is basically a compat issue with this CVS  
> version of OpenMCL in this architecture and/or UFFI. However, I  
> thought I would share this just in case anyone has information that  
> could speed this process or any suggestions of what I may try next.
>
> Thanks,
> Daniel
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel