From scaekenberghe at common-lisp.net  Wed Dec  8 12:03:55 2004
From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe)
Date: Wed, 8 Dec 2004 13:03:55 +0100
Subject: [cl-prevalence-devel] long-chains-bug
In-Reply-To: <F5CDF619-404C-11D9-820F-000A95B3E0D2@common-lisp.net>
References: <5bd1017b04112618583e244d85@mail.gmail.com>
	<F5CDF619-404C-11D9-820F-000A95B3E0D2@common-lisp.net>
Message-ID: <3C52EECC-4911-11D9-AE91-000A95B3E0D2@common-lisp.net>

On 27 Nov 2004, at 09:18, Sven Van Caekenberghe wrote:

> James,
>
> Thanks for the feedback. Here is a first reaction, without looking at 
> your code.
>
> On 27 Nov 2004, at 03:58, James Wright wrote:
>
>> The first bug is pretty minor: An attempt to serialize a
>> standard-object will fail if any of its slots are unbound.
>
> This is probably fixed in the CVS version (among other things) - have 
> a look at the mailing list(s).
>
>> The second bug is somewhat mystifying.  Attempting to serialize an
>> object that points to another object that points ... to another object
>> that points to nothing will fail with a stack overflow if the length
>> of the "chain" is any longer than about 1300.  I'm using Lispworks
>> with a stack size of 64000.
>
> I never tried chains that long. But yes there might be a problem here: 
> the serializer tracks every object it encounters, and is itself a 
> recursive process. Although 1300 doesn't seem that large a number. I 
> am using LW myself, and sometimes you have to expand the stack a 
> couple of times for very recursive code to succeed.
>
> I will have a look at your example later on.

It became a lot later - sorry about that.

First of all, thanks James for the excellent test code, I'll reproduce 
your code with my modifications here:

(in-package :cl-user)
(asdf:operate 'asdf:load-op :cl-prevalence)

(defclass bar ()
   ((slot-1 :accessor slot-1)))

(defun unbound-slots-bug (&optional (fname 
#P"/tmp/unbound-slots-bug-1.xml"))
   (let ((obj (make-instance 'bar)))
     ; Try to serialize an object that has an unbound slot
     (with-open-file (out fname :direction :output :if-exists :supersede)
       (s-serialization:serialize-xml obj out))))

(defun long-chains-bug (&optional (depth 2000) (fname 
#P"/tmp/long-chains-bug.xml"))
   (let* ((obj (make-instance 'bar))
          (cur obj))
     ; Make a long chain of objects
     (dotimes (n depth)
       (let ((new (make-instance 'bar)))
         (setf (slot-1 cur) new)
         (setf cur new)))
     ; Attempt to serialize the chain
     (with-open-file (out fname :direction :output :if-exists :supersede)
       (s-serialization:serialize-xml obj out))))

(defun read-long-chain (&optional (fname #P"/tmp/long-chains-bug.xml"))
   ; Just read the chain back in
   (with-open-file (in fname :direction :input)
     (s-serialization:deserialize-xml in)))

(defun goto-depth (&optional (depth 2000))
   (let ((oddp (oddp depth))
         (evenp (evenp depth))
         (plusp (plusp depth))
         (minusp (minusp depth))
         (zerop (zerop depth)))
     (if (not (or oddp evenp)) (invoke-debugger))
     (when zerop (error 'forced))
     (unless (or zerop minusp)
       (assert plusp)
       (goto-depth (1- depth))
       t)))

(defun goto-depth (&optional (depth 2000))
   (unless (zerop depth)
     (goto-depth (1- depth))
     t))

As I said before the unbound slots problem is fixed already (unbound 
slots are skipped).

The long-chain problem does indeed occur, but is solvable by expanding 
the stack (or starting with a larger stack).

By the way, deserializing is even worse than serializing by a factor of 
4 (there are 2 XML tags per level, and each function involved in the 
recursion has a larger stack frame).

Both serializing and deserializing are inherently recursive problems 
that are handled as such, the deeper the object structures, the more 
stack they will need to execute. I do not immediately see a solution 
(like using a more iterative algorithm). I know that in garbage 
collection there are solutions to similar problems, but there the 
solution is using the datastructure itself destructively as a kind of 
stack, hardly a solution in our case.

What is a bit strange with LispWorks is the (apparent) amount of memory 
used for the stack.

The last goto-depth version above (forced to be non-tail-recursive) 
breaks the default listener stack limit with a depth of 17275 (on LWM). 
The other goto-depth version tries to make the stack frame larger by 
using a lot of (useless) local variables. That version breaks the stack 
with a depth of 789 (about the same depth where the long-chains-bug 
breaks). So adding 5 local variables increases the frame size by about 
20, which could be explained by saying that each local variable takes 
32-bit and all sizes are in bytes. It still feels like a lot to me...

Any suggestions, ideas ?

Maybe we should test these limits on other platforms ?

Sven