Fennel (Lua compatible Lisp) on Norns

orbit_normal · September 3, 2020, 5:39am

Overview

Discovered fennel pretty recently and thought it’d be pretty neat if I could use it for scripting Norns.

Fennel Maiden: https://github.com/pcvonz/maiden
Fennel Matron: https://github.com/pcvonz/norns/tree/matron-fennel

Note: For debugging right now, the Lua script it generates is being logged.

Setup

If you want to try it out here are the steps.

Summary

Matron setup

ssh into your Norns and cd into the norns directory
Add my fork as a remote git remote add fennel_fork https://github.com/pcvonz/norns
run git fetch fennel_fork
Checkout the branch matron-fennel from my fork
git checkout matron-fennel
run ./waf clean && ./waf build in the ~/norns directory.
Grab a copy of the fennel compiler:
sudo curl https://fennel-lang.org/downloads/fennel-0.6.0 > /usr/local/share/lua/5.1/fennel.lua
[optional] If you want to use Fennel in the REPL, add the flag --fennel to matron.sh:
./build/ws-wrapper/ws-wrapper ws://*:5555 ./build/matron/matron --fennel &
Creating a new script with the extension .fnl in Maiden should convert it to Lua and run it like a regular norns script.

Maiden setup

Navigate to the maiden directory cd ~/maiden/app
Run curl -L https://github.com/pcvonz/maiden/releases/download/v1.0.2-fennel/build.tar.gz | tar xz
Refresh maiden

Uninstalling

Uninstalling Matron Fennel

cd into norns and run
Run git checkout main
Run git reset --hard origin/main (This will remove any uncommitted changes, best to do a git status to make sure there aren’t any uncommitted files you care about!)
./waf clean && ./waf build
Reboot norns

Uninstalling Maiden Fennel

Head over to the releases page of Maiden: https://github.com/monome/maiden/releases
Download the tar.gz and extract it
Replace the ~/maiden/app/build folder with the one from the tar file.

TODO:

[x] Run fennel scripts via Maiden
[x] Add syntax highlighting to maiden (in progress)
[x] Add REPL support via matron (???)
[ ] Figure out clean way to have both the regular lua repl and fennel repl available via maiden.

its_your_bedtime · September 3, 2020, 6:00am

Nice! There is also similar project by @ngwese - https://github.com/ngwese/siren

orbit_normal · September 3, 2020, 8:14am

That’s super cool!
The advantage of using fennel though is that creating bindings to the current monome stuff isn’t necessary since it’s completely compatible with Lua. Translating the norns studies have been a breeze.

Also, I got syntax highlighting working in Maiden (still pretty rough).

orbit_normal · September 4, 2020, 3:22am

I’ve been looking into the possibility of running fennel through the Matron REPL, but I’m not well versed in C (or even Lua for that matter).

Does anyone have any tips, pointers, or a general direction on where to start?

csboling · September 4, 2020, 4:17am

Matron evaluates a line of Lua from the REPL with l_handle_line, which interacts with the Lua C API to put the input line on the stack and evaluate it. A lot of the evaluation code is basically the same as the code for the lua repl that’s part of the main Lua distribution.

It might be that the simplest thing to do here is to have a small script or program which turns a line of Fennel input into a line of Lua and feeds it to matron on stdin. Matron already runs with stdin wrapped by the ws-wrapper program which allows interacting with matron’s stdio over a websocket – this is how the maiden browser app and the maiden repl CLI interact with matron, so you would then just need to have ws-wrapper wrap your Fennel-to-Lua translator.

Cool project!

orbit_normal · September 4, 2020, 8:10am

Alright, I managed to get something working with the matron REPL based on @csboling’s advice (thank you!).

Running into some issues now. For example, when trying to write a simple for loop the value that’s returned is nil (instead of printing n):

(for [n 1 5] (print n))

I believe this is happening because the fennel compiler turns that bit of code into this:

for n = 1, 5 do
  print(n)
end
return nil -- <- The problem??

I’m not totally sure.

The second obstacle is the way I’m processing the code from Maiden. I wrote a little Lua script to compile the code:

#!/usr/bin/env lua                                                                                                                                                                 [0/1893]
f = require("fennel")
local lua = f.compileString(arg[1])
print(lua)

And it’s called in C like so:
(pardon the horrendous code, I don’t really program in C. A lot of this was copied )

int l_handle_line(lua_State *L, char *line) {
        size_t l;
        int status;
        char *command = "/home/we/Fennel/compile_fennel_string.lua ";
        char *str = (char *)malloc(sizeof(char)*(strlen(line) + strlen(command + 4)));
        strcpy(str, command);
        strcat(str, "\'");
        strcat(str, line);
        strcat(str, "\'");
        FILE* f = popen(str, "r");
        if (f == NULL) {
                fprintf(stderr, "l_handle_line: Failed at parsing fennel.");
                exit(1);
        }
        char *output = NULL;
        size_t len = 0;
        while (getline(&output, &len, f) != -1)
                    fputs(output, stdout);
...
}

The problem I’m running into is if I pass in something that contains ' then that’s going to break the compilation step because the string of Fennel code will be terminated prematurely. Example:

(print "it's not ok")
is basically like running this in bash:
./compile_fennel_string.lua '(print "it's not ok")'

Not an insurmountable problem, but it feels hacky that I have to deal with it.

Ok, collapsing the generated lua to a single line fixed the first problem
Peek 2020-09-04 01-08

orbit_normal · September 8, 2020, 6:29am

Made some breakthroughs this weekend and a hit another roadblock. I’ll add an update on how to get this running in the original post.

For the REPL: I’m now evaluating the incoming string of fennel through the Lua C API.
Maiden now supports decent syntax highlighting for fennel. Still could use some tweaks.
It’s possible to load both Lua files and Fennel files through Maiden.

I thought it’d be nice to be able to switch the REPL between Lua and Fennel, but I can’t find a decent way to got about this. For now, there is a flag you can add to matron.sh to start the REPL in fennel mode:

#!/bin/bash                                                                                                  
./build/ws-wrapper/ws-wrapper ws://*:5555 ./build/matron/matron --fennel &

In the future I might take what I did in lua_eval.c out into a separate program that can be wrapped by ws-wrapper. I think that’s probably the way it should be done .

Pretty stoked where it is right now.

moogah · September 13, 2020, 3:05pm

This is super interesting, the buzz around lisp has always caught my attention but in the several attempts to get more familiar with it I’ve not really understood why it has this hype as such a good language, maybe some norns apps will give me a change to compare the code and see for myself if and how it offers something more powerful and elegant than a more recent language

lijnenspel · September 13, 2020, 3:48pm

Following this with excitement! Did a tiny bit of Fennel for the Advent of Code last December and liked it!

philomates · May 13, 2022, 12:17pm

This is very cool! I’m a lisp developer whose is interested in trying my hand at some norns scripting, so I decided to give this a try.
I was able to follow your instructions but updating your changes to what is current in the norns main branch (top 3 commits of Commits · philomates/norns · GitHub)

With this I could try out your script to get simple text drawing:

(global init (fn []
  (set _G.color 3)
  (set _G.number 84)
  (set _G.mode 0)))
  
(global redraw (fn []
  (when (= _G.mode 0)
  (screen.clear)
  (screen.level _G.color)
  (screen.font_face 10)
  (screen.font_size 20)
  (screen.move 0 50)
  (screen.level 15)
  (screen.text "cool")
  (screen.update))
(when (= _G.mode 1)
  (screen.clear)
  (screen.move 0 20)
  (screen.text "wild")
  (screen.update))))
  
(global key (fn [n z]
  (if (= n 3)
    (set _G.mode z)
    (set _G.color (+ 3 (* z 10))))
  (redraw)))

I’m curious:

have you continued to use fennel with norns scripting after this initial push?
could you provide more details on how you use the REPL? Like just via the Maiden web UI? I think that maiden.sh went away since you posted (or I couldn’t find it) and I don’t fully understand how the ws-wrapper stuff is launched (referring to your comment: [optional] If you want to use Fennel in the REPL, add the flag --fennel to matron.sh: ./build/ws-wrapper/ws-wrapper ws://*:5555 ./build/matron/matron --fennel & )

and more generally, if anyone has resources on how to hook up a normal editor repl client to send forms to maiden, I’d be very grateful. I tried doing simple things like websocat ws://127.0.0.1:5555 and sending lua/fennel forms over but I think I’m missing something.

EDIT: Just found that the websocat thing works for lua code if you add a --protocol bus.sp.nanomsg.org flag. Now I just need to figure out how to start the maiden-repl thing with the fennel flag enabled in matron (by hand looks like ./build/matron/matron -f 1 but not sure how that works with ~/norns/start.sh)

EDIT II:
I’ve continued playing around with this and wanted to share some results and blocks:

In order to handle the Figure out clean way to have both the regular lua repl and fennel repl available via maiden TODO I added a extended the matron engine with some lua functions to toggle back and forth between interpreting repl commands as lua or fennel (this commit as a part of this dev branch). With this, it the repl defaults to evaluating lines as lua but _norns.enable_fennel() will switch to fennel. ((. _norns "disable_fennel")) then switches back

I was also able to hook up nvim + conjure (repl plugin) to talk with the maiden repl via a config like this

let g:conjure#filetype#fennel = "conjure.client.fennel.stdio"
let g:conjure#client#fennel#stdio#command = "websocat --protocol bus.sp.nanomsg.org ws://norns.local:5555"
;; this is a hack to help conjure know when the full response has been received
;; it is needed because I couldn't figure out how to set a user prompt pattern to websocat:
let g:conjure#client#fennel#stdio#prompt_pattern = "\n"

The open issue I’m playing with now is how to send multi-line expressions to the repl via both the maiden web ui and my editor repl client. It seems like the repl server expects input to be one line and the web ui repl is able to send multi-line forms by replacing \n with ; for lua, which doesn’t work with fennel. I tried to extend the maiden web ui with this change, to allow it to work with fennel too, but I can’t get maiden to build so I couldn’t test it.
Ideally the repl server could handle receiving multi-line input. I wonder if folks have looked into that before