[cffi-devel] Re: WITH-FOREIGN-STRING allocates on heap?

[Luis Oliveira]

"Hans Hübner" <hans at huebner.org> writes:

> On Fri, Jun 20, 2008 at 9:13 AM, Luís Oliveira <luismbo at gmail.com> wrote:
>> I believe we were worried about big strings and the possibility of
>> overflowing the stack. I suppose we could add an argument to
>> WITH-FOREIGN-STRING and :STRING to force stack allocation.
>
> Is this really a realistic issue with modern systems?  Some sampling:
[...]
> So, at least ~10 MB for the 32 bit FreeBSD machine, at least ~1 GB for
> the amd64 Linux box.

Many Lisps seem to operate with much smaller stacks.  Here's the test I
used:

  (defconstant +size+ (* 50 (expt 2 10)))        ; 50 KB

  (defun test ()
    (labels ((ek (n)
               (cffi:with-foreign-pointer (p #.+size+)
                 (loop for i below +size+
                       do (setf (cffi:mem-ref p :char i) 0))
                 (format t "[~A] ~A: total allocated ~:D~%"
                         n p (* +size+ n))
                 (ek (1+ n)))))
      (ek 1)))

  (compile 'test)


SBCL: ~2 MB followed by "the party is over."
CCL: ~2 MB followed by a segfault.
CLISP: 8 MB followed by a graceful stack overflow.
Allegro: ~4 MB followed by a graceful stack overflow.
(used 40 KB chunks otherwise Allegro refused to do stack allocation)

This is on linux/amd64 (but I think my copy of Allegro is a 32-bit
version).  Hopefully my test is not (too) bogus.


> Given this, I'd think that making stack allocation be the default and
> heap allocation an option would be beneficial.  If you really don't
> like it, making the default a compile-time option would be better than
> mandating an allocation policy, I think.

Since the size argument has to be known at compile-time (IIRC), maybe
WITH-FOREIGN-POINTER can look at that size and decide if it should
fallback to heap allocation.

CCL and Allegro do this.  SBCL and CLISP don't.  If we did that check
ourselves we'd get more consistent behaviour across the various Lisps we
support and that might reduce the likelihood of overflowing the stack
with big objects, particularly strings.  Also, it would be nice if SBCL
and CCL could handle stack overflows without crashing.

-- 
Luís Oliveira
http://student.dei.uc.pt/~lmoliv/