I’m developing a puzzle/strategy game using a custom engine, a project I work on for fun in my free time. For this, I’m using Jai, the programming language created by Jonathan Blow. It’s been a great opportunity to explore Jai’s features, and I particularly appreciate its lack of bloat and fast compile times. Its metaprogramming system is the most comprehensive I’ve ever used – and likely the most complete among system programming languages.

This series of posts is about how I (over?-) used the metaprogramming system. This post gives an overview and a most basic examples. In the next one we will see the more complex ideas.

So here’s a screenshot of my prototype:

The interface is heavily inspired to Blender’s Shader Editor. It is composed by Cards, which are the purple elements like Generator or Crafter, each of them exposing some Pins that can be connected to move Items (the alpha’s and beta’s) between Cards.

Cards can be manipulated by the player in different ways, at different levels of uniformity between Card Types:

• Some operations are independent by the Card Type, for example dragging a Card around the map. All Cards are just rectangles with a title and some stuff in it after all.
• Some operations are slightly dependent by the Card Type, for example the Pins have the same behavior on all the Cards, but each Card Type has a different set of Pins to keep track of.
• All the other operations must be heavily dependent on the Card Type. A good example is the function that controls the behaviour of the cards on its Items.

The idea of metaprogramming usage is to handle each of this cases with the appropriate degree of automation.

A Step Back: an Overview of Jai Metaprogramming

Quoting Wikipedia, “Metaprogramming is a computer programming technique in which computer programs have the ability to treat other programs as their data”. Thus metaprogramming may allow computer programs to read the inner structure of computer programs, and modify it.

Jai allows that in a few ways:

Compile Time Execution. Well, maybe this is not strictly speaking a metaprogramming tool, but I’m including it in this list as first entry because the ability to perform meaningful operations during the compilation process is an important part of the system, as we will see.

This can allow funny things such as the Jonathan Blow’s demo of a program that throws a compilation error if you are not good enough at space invaders.

Reflection. Types are first-class citizien in this language, and the compiler provides a procedure to obtain informations on a type, both at compile time and execution time. Is my type a numeric type? How many bytes is it large? Is it a struct? If so, which are its fields? And their types?

There are also other reflection utilities, such as code_of that returns the definition of an identifiers, #exist that check if an identifier is defined and others.

Code Insertion. While compiling a directive like #insert can be encountered with a string argument. When it happens the string is baked directly into the code. The interesting part is that given the capabilities of the compile time execution, the string that is being inserted can be the result of a computation.

Macros. Aka procedures++, will probably see a rework in the future. These are defined in similar ways as functions, and can use identifiers from the scope where they are invoked, and add identifiers to that scope.

The Metaprogram. This is actually the nuclear bomb of metaprogramming.

1. We start compiling a first program, running jai first.jai, where first.jai is the a jai source file.
2. The program in first.jai can tell the compiler “I don’t want to be actually compiled I am here just for the compile time execution”.
3. After that first.jai may ask “Dear Mr. Compiler, here a file main.jai, can you compile as a new program? Please keep me informed of how it goes”.
4. “Oh sure! In main.jai I have seen that the programmer has created this procedure and a couple of structs. Here’s their AST”
5. “Uhu. Thank you! Based on what I have seen in the function, can you please add this string to the source code of the main.jai program?”

and the conversation can go on in a cycle of back and forth between the compiler that parses and typechecks all the code, and the metaprogram in first.jai that issues new code based on the result. This cycle ends when everything is compiled and no new code is issued, or terminates with an error if there is a compilation error.

A Warm-up Use Case: Log Functions

We have a log function that along the error wants to print the stack trace at the point it was invoked. However we don’t want the stack trace to be too long as it would be unreadable, especially if multiple errors are logged. Thus we want to have some functions that won’t be included in the stack trace.

This is a typical function declaration:

i_dont_want_to_be_logged :: () {
    // code here
}

We require a stack trace such as

procedure_a                filename1.jai:34
procedure_b                filename2.jai:156
i_dont_want_to_be_logged   angry_file.jai:21
procedure_c                filename1.jai:3

to be reduced to

procedure_a                filename1.jai:34
procedure_b                filename2.jai:156

A print stack trace code would be in a vanilla state as

for entry : stack_trace {
    print_entry(entry);
}

Easy enough. Now it would be helpful to have an array of function names that should interrupt the trace printing.

log_interrupting_functions :: string.["i_dont_want_to_be_logged", "and_so_do_i"];

Now we can rewrite easily the print code as

for entry : stack_trace {
    // exit if we find an interrupting function
    if array_find(log_interrupting_functions, entry.name) then break;

    print_entry(entry);
}

This would require the interrupt_trace_functions to be updated each time we want a log-interrupting function, and moreover if multiple libraries are used all of them should interact directly with the logging code. We can however use the metaprogram to give a function an “interrupt-log” mark, and build the array in an automatic way.

The mark is carried through the notes, which are strings that Jai allows to to attach to sever AST nodes, and may be used by a metaprogram. We attach a log_interrupting note with

i_dont_want_to_be_logged :: () {
    // code here
} @log_interrupting

Let’s move on the metaprogram part. This code is run at compile time by first.jai.

// this is a dynamic array.
log_interrupting_functions :: [..]string;

is_log_interrupting_functions_array_inserted := false;

compiler_begin_intercept(w);

while true {
    message := compiler_wait_for_message();

    if message.type == {
      case .COMPLETE;
        break;

      case .TYPECHECKED;
        message_typechecked := cast(*Message_Typechecked) message;

        for head : message_typechecked.procedure_headers {
            notes := head.expression.notes;
            for notes if it.text == "log_interrupting" {
                array_add(*log_functions, head.expression.name);
                break;
            }
        }


      case .PHASE;
        message_phase := cast(*Message_Phase) message;

        if message_phase.phase == .TYPECHECKED_ALL_WE_CAN
        && !is_log_interrupting_functions_array_inserted {
            is_log_interrupting_functions_array_inserted = true; 

            // this function should return a string like 
            // "log_interrupting_functions :: string.[
            //     \"i_dont_want_to_be_logged\",
            //     \"and_so_do_i\"
            //  ];"
            // It is a not very interesting function that works on regular strings.
            string_to_be_inserted := build_array_definition(log_interrupting_functions);

            add_build_string(string_to_be_inserted, w);
        }
    }
}

compiler_end_intercept(w);

A few comments on this code. The conversation back-and-forth between first.jai and the compiler is enclosed between compiler_begin_intercept and compiler_end_intercept.

The compiler sends data to the metaprogram through compiler_wait_for_message. It gives messages of different kind. COMPLETE is used when compilation has ended, and we are prompted to exit the loop. TYPECHECKED returns all the things that have passed the typechecked phase, and we are returned with their definition. We use these messages to gather all the log-interrupting functions.

With TYPECHECKED_ALL_WE_CAN we can be sure that we don’t miss any function we want to gather. Notice the is_log_interrupting_functions_array_inserted guard. It’s here because after inserting the new array definition, we are going to have another round of TYPECHECKED messages and after that a new TYPECHECKED_ALL_WE_CAN, and we don’t want do define the same identifier twice.

Inject the Git Hash

Another useful usage of the metaprogram is very simple, and can be summarized in a couple lines of code in the metaprogram:

version := get_version_string();
add_build_string(tprint("VERSION :: \"%\";", version), w);

The function get_version_string can launch a new git process for querying the current commit hash and the last tag on the current branch. These strings can then manipulated by leveraging the full Jai language, and not some external scripting language.

After that the constant VERSION will be available in the regular program.

I believe this is a winning approach over more complicated (because it requires programmers to be proficient in two or more languages) and less powerful alternatives like CMake and similar tools.

Conclusion

In this article we have seen:

• Basic ideas in what metaprogramming is;
• Some simple examples of metaprogramming usage;

In the next post I’ll write about a more complex usecase.

Thanks XKCD for the humor. Thanks Jonathan Blow for the compiler.

Comments can be submitted on Hacker News. DMs at my contacts available at my Home Page are more than welcome!

With love,

– Alessio