[iterate-devel] PATCH: support (finding (values ...))

[Max Mikhanosha]

Hi,

Well I did not wanted to request a feature/wish-list item without first
trying to solve it myself; so included patch was more to start the
discussion and wasn't intended to be a finished product. Sorry I did not
made that clear in the submission.

Now to the subject matter itself (citations re-arranged to have
discussion in a more logical order, with the response to patch
critiques in the end)

> Iterate claims to be an extensible iteration facility. Thus I wonder
> why many contributors in recent years have spent their time
> understanding and modifying the low-level implementation, instead of
> using Iterate's documented extension mechanisms?

The reason number one is that I got intimidated by the reader macros
part, ie I've seen sharp-L and bang-vars thing, and thought too myself
"well this seems to be pretty complicated thing right there".

Another reason is that even as you said implementing this with
defmacro-clause does not allow (finding-values (values (the fixnum a)
(the fixnum b)) type of expression, that would make "MAX" variable
type known. Since my whole reason for wanting this functionality was
optimizing my inner loop code for speed, that would defeat the
purpose.

> Comments are welcome.

>  - general mechanisms for multiple values?

Should be part of the base library (IMHO of course)

What attracted me to iterate in in a first place is ability to do
"finding (list a b c d e) maximizing (f a b c d e)" type of clause,
and ability to execute it from inner loop.

Finding x,y,z maximizing f(x,y,z) is much more common (well at least
in mine) AI/GA/optimization type of code then finding single values.

So while "finding" a one clause among many (iterate (collect/finding))
is probably 80% of all my usage of iterate, so making "finding" more
efficient and easy to use is a priority.

>  - better API for gatherer into the default value(s) vs.
> iter::*result-var*?

My *result-var-2..9* business was silly I realize that now. But one
can change the main (iter) macro so that it does:

(if *result-var* `(values ,@(ensure-list *result-var))
  nil)

And then making it so that when the clause returns multiple values it
does something like:

(if (listp *result-var*) ;; other clause already returned (values)
    (unless (= (length *result-var*) ...num-values-being-returned...)
      (error "Previous clause(s) returned different number of values"))
    ;; else
    (cond ((have-binding-for *result-var*)
            (error "Previous clause(s) returned single value"))
          (t ... convert *result-var* to list and add additional vars here ...)
        
>  - exact prefix bug caretaker?

don't know enough about this. I assume code that parses this should
have some kind of look-ahead added to solve ambiguities like that? I'm
not volunteering tho.

Regarding the critiques of the submitted patch, some of them I can
solve, let me know if you want me to work on it for a few days with
the goal of inclusion. For my own uses I'm pretty happy with a
quick-fix I got.

>  - Generality: This syntax (VALUES ...) pattern based extension
>    does not cover functions returning multiple values
>    e.g. (finding (floor x y) maximizing (foo x y))

This can only be fixed by differentiating on syntax. While I do not
believe that finding clause like above would be used very often, I
agree it should be supported.

Syntax variants that can be used (note we have to know the number of
values being returned by the function):

- finding (values (foo x) (bar x))   ; returns 2 values

vs

- finding (n-values-of 2 (foo x))    ; returns 2 values returned by foo
or 
- finding (2-values-of (foo x)))     ; will have to parse symbol name

Also the following shortcut will have to be supported:

- finding (the fixnum (values ....)) 

being same as enclosing each individual value in fixnum, otherwise
code looks too heavy with relatively many values like >4 being
returned.. LOOP has similar shortcut (loop for (a b c) fixnum) and its
very useful.

>  - Arbitrary implementation limit to 9 *result-vars*
>    (MULTIPLE-VALUE-LIMIT is the only one that would make sense,
>     but it's not practical)

Yea that was a thinko. Simple making *result-var* a list of var names
can be used as a flag that one of the clauses returns (values) which
gets rid of 9 vars limit and reduces number of lines changed.

>  - A lot more complexity added to the source for just one clause.

Its just one clause but not all clauses are created equal. IMHO
finding is one of the most important iterate clauses, along with
collect, so should get correspondingly more attention. 

I can offer some fix for this by eliminating 9 additional result-vars
ugliness, also some of the changes were really cosmetic ie I renamed
the variables in the function var to vars, expr to exprs etc to
reflect the fact that these were now lists.

>  - Unclear yet whether this syntax extension makes sense for other
> clauses.

collect (values a b c) returns 3 lists? 
collect (values a b c) into (values alist blist clist) collects the same 3 lists into 3 variables?

;; convert a single list of something complicated that have 2
;; attributes into 2 lists of each attribute

(iter (for elem in complex-list)
      (for a = (extract-a elem)
      (for b = (extract-b elem)
      (collect (values a b))))))

>  - &optional INTO is not integrated. What would it look like?
>    INTO (values a b c) INTO ((values a b c) max) INTO ((a b c) max)

That one is easy to fix, I just forgot to do it. It will look like so:

INTO ((values a b c) max), ie element of least surprise. You just use
(values..) form in the same place as the single variable in 1 variable
case.

>  - Slight bug: clause as expression must define result and
>    constantly yield either all, or only primary value, e.g.
>    (for (values running-x running-y) = (finding ...))
>    [if that were the only thing, I'd fix it myself before applying the
> patch]

Missed that one also, shoulde be fixable as already noted.

Regards,
  Max