There is nothing cooler than a macr- Err... Mixin?

Posted on by Idorobots

One of the most distinctive features of Common Lisp and Lisp in general, are its code-generation and code-manipulation capabilities.

Probably the best example is the LOOP macro - a Swiss Army knife of iteration that can do pretty much anything. The following snippet iterates a list of random numbers collecting some statistics of its contents and does that while being very concise and readable:

(let ((random (loop with max = 500
                    for i from 0 to max
                    collect (random max))))
  (loop for i in random
          counting (evenp i) into evens
          counting (oddp i) into odds
          summing i into total
          maximizing i into max
          minimizing i into min
        finally (format t "Stats: ~A"
                          (list min max total evens odds))))

Stats: (0 499 120808 261 240)

Here's my version written in the D programming language. It uses CTFE to transform LOOP definitions and generate D code at compile time while mixing it together with the rest of the code extending the language with a DSL of similar capabilities:

auto random = mixin(Loope!q{
    with max = 500
    for i from 0 to max
      collect $​$ uniform(0, max) $​$​
});

mixin(Loop!q{
    for i in random
      counting $​$ (i&1) == 0 $​$ into evens
      counting $​$ (i&1) == 1 $​$ into odds
      summing i into total
      maximizing i into max
      minimizing i into min
    finally $​$ writeln("Stats: ", [min, max, total, evens, odds]) $​$​
});

Stats: [2, 499, 126399, 229, 271]

To compile this, D's compiler first parses LOOP definitions passed to the Loop template using a parser generated (optionally at compile-time ;)) by Philippe Sigaud's Pegged parser generator. After that it traverses the parse tree translating it into valid D code, resulting in this, full of type-inference and meaningless identifiers, monstrosity:

auto random = (() {
    auto max = 500;
    auto __i_0 = max;
    auto i = 0;
    typeof(uniform(0, max))[] __accumulator;

    for(;;) {
        if(i >= __i_0) break;

        __accumulator ~= uniform(0, max);

        i += 1;
    }
    return __accumulator;
})();

{
    auto __i_0 = 0;
    auto __i_1 = random;
    auto __i_2 = __i_1.length;
    typeof(__i_1.init[​0]) i;
    uint evens = 0;
    uint odds = 0;
    typeof(i) total;
    bool __max_3 = false;
    typeof(i) max;
    bool __min_4 = false;
    typeof(i) min;

    for(;;) {
        if(__i_0 >= __i_2) break;
        i = __i_1[__i_0];

        if((i&1) == 0) ++evens;
        if((i&1) == 1) ++odds;
        total += i;
        if(!__max_3 || i > max) {
            max = i;
            __max_3 = true;
        }
        if(!__min_4 || i < min) {
            min = i;
            __min_4 = true;
        }

        ++__i_0;
    }
    writeln("Stats: ", [min, max, total, evens, odds]);
}

That's more than three times the volume of the previous snippet and it's not nearly as readable. Imagine writing it yourself each time. The real fun, however, starts with complex loops:

auto random = randomStuff();  // Implemented elswhere. ;)

auto result = mixin(Loope!q{
    initially $​$ writeln("Loop test:"); $​$​

    with isEven = $​$ (uint x) => ((x&1) == 0) $​$​
    with updateAnalysis = $​$ // A D function analysing our data.
                             (uint[] stats) {
                               static count = 0;
                               if(count++ % 10 == 0)
                                   writeln("Analysis: ", stats);
                             } $​$​

    with max = 500
    with data = $​$ // Yo dawg...
                  mixin(Loope!q{
                      for i from 0.0 to max by 1.337
                        collect i
                  }); $​$​

    for i from 0 to max
    for datum in data
    for r in $​$ sort(random) $​$​

      if $​$ isEven(i) $​$​
        minimize i into minEven and
        maximize i into maxEven and
        unless $​$ i % 4 == 0 $​$​
          sum i into evenNotFoursTotal and
          collect datum into floats
        end
        and sum i into evenTotal
      else
        minimize i into minOdd and
        maximize i into maxOdd and
        when $​$ i % 5 == 0 $​$​
          sum i into fivesTotal and
          collect r into randoms
        end
        and sum i into oddTotal

      do $​$ updateAnalysis([minEven, maxEven, minOdd,
                            maxOdd, evenTotal, oddTotal,
                            evenNotFoursTotal]) $​$​

      finally $​$ writeln("Floats: ", floats); $​$​
      finally $​$ writeln("Randoms: ", randoms); $​$​

      finally $​$ return [minEven, maxEven, minOdd,
                         maxOdd, evenTotal, oddTotal,
                         evenNotFoursTotal]; $​$​
});
writeln("Result: ", result);

Just as the Lisp' LOOP macro (not as elegantly, though) it blends together really well with the host language allowing for arbitrary D code to be used inside of the loop (including Loop template itself).

The code generator, however, is very different to any Common Lisp LOOP implementation. D being a statically typed language with complex, unlike Lisp, syntax lacks symbol manipulation and quasiquote, meaning it has to rely on string processing, CTFE and string mixins that, despite being highly experimental and prone to performance issues, are still very powerful.

Also... There is a macro keyword reserved for future use in the language.

Fingers crossed for this one. ;)

The title refers to this video.

2016-02-16: Adjusted some links & tags.