Interview with oofoe about REBOL, Forth, Decker, Janet and the VFx Industry

Oofoe (sourcehut) is a REBOL-refugee comes from animation and now works on precision instrument software. He makes puzzles in Decker and games in Janet (using his Janet->JavaScript transpiler) and Racket and Racket and, again, Racket and Racket and even visual novels. In this interview, we discuss: DSL design exotic languages (REBOL, Forth, Janet) Decker industrial domains (VFx) Do you know of any resources to set up additional targets in %project.janet? This can be good, but what's the %? Oh, sorry... The % business is a holdover from REBOL that I took up with. They indicate file paths to the parser with a leading percent sign. So you can say things like write %survey.csv data and you don't need quotes around the path like most other languages. REBOL had something like 20+ types that the parser understood. REBOL was my first introduction to PEG (which led me to Janet...) Oh! Please shill REBOL! This is now an interview. REBOL had four things really going for it (in my opinion): A very simple, concise syntax, Built-in, first-class PEG (parsing expression grammars), An AMAZING declarative GUI library, And, between the docstring features and decompiler, it was self-documenting. Here is a digital clock application with GUI in REBOL: view layout[origin 0 t: h1 red black (to string! now/time) rate 1 feel[engage: [t/text: now/time show t]]] Now, that's a purposely code-golfed example, but it gives you an idea of how small you can go. But you can go big too -- I taught myself the fine points of the GUI that weren't covered in the manual by decompiling the VID dialect and pulling on the levers thus revealed. I then used it to build (among other things) a visual video editor, tools to ingest and process stuff into a professional editing package, and lots and lots of utilities that could have nice interfaces because they were so easy to do in REBOL. This is a simple visual flow-based compositor that I did for a contest at work a while back: https://hg.sr.ht/~oofoe/tc Unfortunately, despite its incredible capabilities and genius antecedents (designed by Carl Sassenrath, who did the Amiga operating system), REBOL was eventually discontinued. So I found myself looking at other languages. Janet has surpassed REBOL in the PEG category because of how it handles the grammar. And it's not too far behind in the self-documenting sweepstakes. Syntax... Well, Janet's a Lisp and REBOL took inspiration from SmallTalk and Forth. Unfortunately Janet has no "decent" GUI bindings yet. Probably a difficulty of making threading work? I once put together a REBOL style declarative GUI package for Python, so have considered doing that for Janet with RayLib. But haven't gotten around to it. Why did Sassenrath quit? It was tragic that Carl Sassenrath (the guy who invented it) decided to quit (and run a vineyard) before taking it to 64bit, but on the other hand, maybe it was the right decision, because his idea of "programming in the small" had a hard time finding a receptive audience, even though it was spot on. It also didn't help that his company hoped to charge for a higher-spec professional version of REBOL during the era when free Python, Ruby and JavaScript were stomping around the world in seven-league boots. I really loved REBOL, though, for me it just worked. Programming in the small? Wikipedia has a bit about it, but here's his manifesto. "Programming in the small" is the idea that programs don't have to be big and complex to be useful and robust. Part of that philosophy was achieved by having lots of useful data types in REBOL (e.g. 16:34 is a time, 2021-05-19 is a date, 34x994 is a coordinate and so on. You didn't have to do anything special (like import libraries), you could just use them. The other part is stuff like "global variables are good enough" and not worrying too much about side effects. You're making a small program that does a limited set of things, it doesn't have to be a big interoperating thing (like anything in Java), or have lots of ceremony, like C++. I favour human-sized code bases. Giant edifices of code are rarely justified (or justifiable) and often obviated by reframing the questions you're trying to answer. (Yes, there are exceptions -- a state of the art optimized graphics renderer probably won't be less than 500 lines (ha!) and if you want a feature-complete reimplementation of Micro$oft Word, well...) However, even when doing somewhat complex things (render farm control software, multi-user studio management system, etc.) I try to make the data control the flow -- the code itself is fairly simple and "obvious" once you've seen how the data is organized. I am greatly informed by Forth. "Proper" Forth development consists of creating the vocabulary required to solve your problem, but you don't have to throw away the rest of Forth. In Forth, a DSL is basically an emergent property of the language -- you have all the same tools as the language implementer at your disposal. Things do generally have a "forthy" feel to them when you're done (although some people have implemented FORTRAN, BASIC and even C in Forth...), but that's generally no defect when you're working in an environment that supports it. For whatever reason, DSLs in other languages seem (to me) to require an extra mental step because they're divorced from the language proper. Like, say, Racket. You can do other languages and syntax in it, but it's such a bother. That said, most of my Forth programs have been fairly small by the standards of ray tracers or word processors. My render-farm controller was Forth-based, but it had a fairly simple job once I re-framed the assumptions. I suspect I would approach word-processing the same way -- i.e. it wouldn't look like MSWord. ; - ) I used to joke that all the programs I've written in my career would fit on a single 1.44MB floppy. I don't think that's true any longer, but I try. Forth also likes global variables. Chuck Moore said local vars are dangerous. Forth uses the stack for what people would typically use scoped function local variables for. E.g. def add(a, b): return a + b in Forth would be : add + ;(where : starts a function definition and ; terminates it). Where's the variables? That's the thing in Forth -- they're put on the stack by whatever calls add and it immediately operates on them, so in many cases they're simply not needed. So, he's not wrong... They're definitely an extra thing that has to be managed. However, he's also the only person I know of that could implement an MPEG decoder without them! For lesser mortals, sometimes they're useful. I knew Forth before I took up with REBOL, but REBOL really made it clear how a concise notation and rich data types were worth an extra 30 IQ points (paraphrasing Alan Kay). That's one of the things that really frustrates me with "modern" software development. REBOL shows the way to a world where you can program interesting things quickly, almost like building stuff on the fly with your (say) C64 back in the day with the built-in BASIC. To me, that's "vibe-coding", not the AI thing. Just starting with a blank screen (or editor buffer) and seeing where it takes you. The Jester transpiler I did for Janet was kind of that -- I had the germ of an idea, "could I translate this Janet to JavaScript with a PEG?" And then I started adding stuff to it until I could write whole programs! Janet's not REBOL, so there are a few parenthesis mixed in, but it worked very well -- almost stream-of-consciousness. (The code reflects that, of course, I keep meaning to clean it up a bit, at least for error checking.) How did you discover Forth? I had my first computer, that I bought with all the proceeds from my summer job -- a sleek black plastic Sinclair ZX81. The ZX81 was an interesting machine, very compact (and cheap) and more capable than it had any right to be. from what I understand its BASIC even influenced Python to some extent. However, while it was a clever design, it was slow. Like really slow. There was a thing called "PluriForth", which was a replacement ROM chip that turned it into a complete multitasking machine. But it was a hundred bucks -- way out of my price range. But I could get ZX-Forth, which came on cassette. It had a number of sharp edges and was the exact opposite of user-friendly. But I found Leo Brodie's "Starting Forth" in the library and things got better from there. The thing that absolutely sucked me in with Forth was that, unlike BASIC, it wasn't a black box. You could pull the pieces apart and see what was going on. What was that render-farm controller you mentioned? In Visual Effects (VFx), you describe a scene to a computer and then move a virtual movie camera through it. Twenty four times a second, for the number of seconds the camera is in play, you have the computer generate a single image, or "frame" (named for the black bordered areas of motion-picture film). Put together, they make an animated sequence. This process is called rendering. It can be terribly time-consuming, but fortunately it's almost trivial to parallelize if you have a network of machines dedicated to rendering and the software to control them. This is a render-farm controller, and once upon a time, it was very, very expensive. Back in the day, being able to dynamically distribute computer jobs to disparate processors was a big deal. You only needed that sort of capability if you were doing A-bombs, serious weather sims or... VFx. But, one night, while sleepily hand-feeding a handful of computers render jobs for an IMAX project, I started thinking about how I could do it better. A friend of mine was working in California at one of the big studios. He told me horror stories about how different projects would have "render wars" to try to get more compute to get their images out before deadline. They would hack other user's accounts and de-prioritize (or delete!) their jobs, change the size of their render pools so they could "steal" more machines, that sort of thing. I decided I wasn't going to have any of that, thank you... Instead of the traditional job prioritization and pooling, on my render farm you could ask for any number of machines and give your jobs the highest possible priority, no hacking required. HOWEVER! If you asked for more machines than another project, their jobs got priority over yours. Say we have 100 machines. Project "A" might ask for all 100 and run the jobs through at mid priority. However, project "B" could ask for only 20 machines and even at lowest priority, they would get red-carpet service. Now, once "B" was done, "A" would get everything to itself. There were some other wrinkles, but the idea was to reward people for asking for a reasonable amount of resources instead of trying to glom them all. It had some other notable features. Most render-farms I had seen at that point were "command-and-control". The render server would contact free machines and start jobs directly, and keep track of each one itself. This usually used a mechanism like rsh or something. It did not scale well. I came up with a client-server architecture where the controller machine (server) just kept a list of available jobs and clients would contact it and request something to do that matched their capabilities. I know this is so trivial today -- it's like I'm describing how to flip a light switch! -- but then, I was standing the established order on its head. I called the software RiCk (render client) and RiSa (render server). I wrote them in Python, with a twist. The server ran a simple Forth implementation that the clients would connect to and issue commands to a REPL. I could also telnet to it and debug or even rewrite the queuing algorithm with hot code updates. The server also handled http -- the work queue was controlled by a web interface that you could connect to with a browser over the same port! At its height, I think it managed around 600 machines. It served us for several years and certainly saved a chunk of licensing fees. Eventually, it got replaced by a commercial solution that had a better sales team than myself... You've used Racket, NewLisp and Janet, how'd you get into Lisp? A long time ago, I worked in civil engineering. Technical drawings and some light design work. I started out doing drawings with precision ink pens on poster-sized semi-transparent mylar sheets, but eventually the company got with the program and set us up with AutoCAD. AutoCAD was primarily architected by a super smart guy named John Walker and he came from a very traditional computer science/mainframe background (as in UNIVAC mainframes). He was very interested in embedded scripting languages (his seminal ATLAST was absolutely formative in my development as a programmer) and pushed to integrate a Lisp interpreter (XLISP) into AutoCAD so users could add their own functionality. Unfortunately, there was almost no documentation in the main AutoCAD manual. Just a half page blurb with a short demo program (written in the Don Lancaster style). I bounced hard off that and figured I'd never have anything to do with Lisp ever again. However, much later, I found Paul Graham's essay about Viaweb, the e-commerce startup he helped found. They programmed using Lisp. I wasn't that impressed with his claims for flexibility or expressiveness (after all, I was already using Perl, right? ; - ), but some of the capabilities he described (closures!) sounded like black magic. Much like when I was paddling around the shallows of AutoCAD and my kicking foot had smacked into the rugose body of XLISP, I wanted to know more. I found CLISP and CMUCL and others in short order. I was ready to unlock all this cool Lisp stuff. I did actually do some stuff with it, like my studio's website and a few utilities. Unfortunately, I just couldn't make Common Lisp "click" for me. The underlying concepts were elegant, but it seemed to be all strangled by no end of janky cruft that seemed to do the same thing ten different ways, but with no road map to picking which to use. And, let me tell you, the Common Lisp HyperSpec was no help at all. But, I got Emacs working and I started figuring out elisp. And I also ran across the Toronto Lisp Users Group. And then... I found NewLisp. NewLisp had some warts and for some reason a lot of people got upset that it called itself a "lisp" when it didn't meet some particular criteria for lisp-dom. However, for me, it was a total breath of fresh air. Things were named rationally and purposefully -- I could often guess the name of a built-in function instead of needing to look it up. There were some great usability features, like implicit structure accesses. And it had a cross-platform GUI library. But most of all, it had excellent documentation! NewLisp was a launchpad. It wasn't perfect, but the language itself made sense and I finally had good, coherent documentation. Among other things, every function in NewLisp had an example program you could just type in and run. This, more than anything, gave me a jet assist when learning. I wish more language authors would understand this. I took NewLisp pretty far. My first "useful" GUI program with it was an aviation weight and balance calculator, which helped with the tedious calculations before a flight: https://hg.sr.ht/~oofoe/wb. I also experimented with a Twine-like 2D writing tool (https://hg.sr.ht/~oofoe/storytrace) and an implementation of Jef Raskin's computer programming teaching platform (https://hg.sr.ht/~oofoe/flow). But there were a few flies in the ointment. One, there wasn't a good way to package (compile) a program. And two, if you wanted a GUI, you needed to have Java. As Oracle began "altering the deal" on Java, this was becoming less and less attractive. So, I started looking for a new place to land. I had played with DrRacket before, but always kind of dismissed it because I didn't really get along with the IDE. It could certainly do some powerful things though. However, in 2018, I finally bit the bullet. I had found out about the Lisp Game Jam on itch.io and I wanted to participate. Since I still didn't have a way to "compile" anything with NewLisp and I wanted to make an executable, Racket looked like the best choice. The experience wasn't perfect, but I actually made it to the end of the jam with a complete game: Castle Monsterous. Small steps, but it worked and I could ship it. I moved all my lisping to Racket. I used it almost exclusively for my recreational (and some work) programming for the next several years, learning about various arcane facets of the language. The "universe" simulation/game framework was instrumental in shifting my thinking to where I could envision my "Canned Heat" not-a-game-engine, and the language creation facilities of Racket led me to my next step -- Urlang. Using it, I made the most polished game I've done yet -- "Class 6 Goes to Hell" which kind of rips off "Helltaker" (just based on a let's-play I saw of it). Now, Racket could compile, but I was starting to get dissatisfied with what it compiled. I was running Windows, but a lot of people in the game jam weren't. So unless they compiled from source, they couldn't play my games. What was available to almost everybody? A web browser. JavaScript was supposed to descend from Lisp, there should certainly be a way to do Lisp on the browser, right? The answer at the time, was not so much. There were some things that looked promising, but most of them had a really high tool cost -- other runtimes, other compilers, Node (shudder!) and so on. Then, finally, I found what I was looking for, almost right on my doorstep. Urlang is a Racket package that compiles Lisp to JavaScript. It uses all the standard Racket compiler machinery (including macros) and you can edit with the Racket IDE. My next several games, including my best ever placing, were all done this way. I'm most proud of zode-tripper, the first game I did with sound and music (working with some very talented Brazilian musicians). It was also where I first started coming some understandings about how to architect games. This was good for jams and I was really excited by the expanded reach, but not so great for all-round programming. After a brief investigation of Julia, I ran across the Janet. It ticked a bunch of boxes for me -- it had some features, like implicit structure access, that I sorely missed from NewLisp. It had built in PEG parsing, like REBOL. And it was lightweight and you could compile an executable. I've now used Janet for a number of interesting projects. Among these are bog-standard file and text transformations, a GUI-based art training tool, network utilities, device simulators and even a scientific/industrial monitoring and control tool. That last has now been released as a commercial product. One of my more recent projects was using the Janet PEG parser to build another Lisp->JavaScript transpiler. From a forum post I made about Janet: With multiple network connections, GUIs and long running processes, multitasking of some sort is a must. Janet provides several flavours. The simplest is a "fiber", which is a kind of in-process green thread. Then, threads (of course), and finally actual multiple process synchronization. So far, I've only needed fibers, but it's been more than enough to handle multiple data streams for simulating and controlling scientific instruments. And compiling. If you're on Linux (or Mac, I think), it works like falling off a log. If you're on Windows it's annoying (thanks Micro$oft), but still doable. It uses a two stage compilation that allows you to control what gets compiled in your source (like C macros, but much nicer). Compared to Racket (or even other scripting language compilers, e.g. Nuitka?), it produces a tiny result (1.7MB for a commercial product where I compiled in all the resources and even the user manual to a single executable). - https://newlispfanclub.com/index.php?topic=5295.msg25390#msg25390 So, that's where I am with Lisp. You've used Decker quite a bit, how have your experiences with it and been? Although I have poked at various array languages, none of them have really stuck with me. However, Lil in Decker was fairly convenient to work with -- I appreciated that it offered english words and a fairly conventional event-driven architecture. I really like the ideas of array languages, but I often seem to trip over their idiosyncrasies, Lil kept things manageable. I always regretted missing out on Mac HyperCard (I was totally PC). I tried a couple of the successors that I could run, but they were disappointing. For instance, one of the big features of HyperCard for me was built-in drawing tools and the clones just expected you to use PhotoShop instead. Decker is the first one I've tried that captures the magic for me -- it's small and fast, has the drawing tools and is extensible -- you can write scripts in Lil to add new features. Once I figured out how to persist data (in a control) things started to fall into place. Decker may not be what you want for "serious" development, but for quick sketches or simple applications it works great! I've used it for pixel art, GUI mockups and, of course the odd little game. The great thing about is that you don't have to use all of it. It's fine as a sketchpad or little language environment to try things out. Also, web-based, so easy to distribute whatever you come up with. How did you get into VFx? Thanks to a TV special I saw about the making of "Tron", I was pretty interested in the cinematic possibilities of computers. So, almost as soon as I started working with AutoCAD, I wanted to do a movie with it. I experimented with AutoCAD and early PC paint and slideshow programs, but I didn't really get anywhere until I got myself hired by a fledgling computer graphics firm started by a Hollywood expatriate. He'd brought over a chunk of surplus iron (DEC VAX 11/750 and a Silicon Graphics 3130 workstation) and set up to serve the Mexican market. He put a help-wanted ad on a BBS (early version of Reddit... ; - ) advertising for someone with art, CAD and programming experience. I took a tour through his basement facility (all the giant computers were installed in his actual house so he was working from home before "wfh" was even a thing) and I was hooked. Within a week I was working on an exploding toothbrush for a tv ad for Colgate Palmolive. The coolest thing about VFx is that cheating is allowed. You animate "to the camera", meaning that the only thing that counts is what the viewer sees, not how you made it happen. You can commit what would be unpardonable sins in any other discipline that valued probity and correctness. But with VFx, you're frequently doing the impossible (a man can fly!?) under impossible time and budgetary constraints. Cheating is the law! Could you talk more about the IMAX project? So, I needed some class credits to graduate college. A friend mentioned in passing that a nearby planetarium he worked with was interested in new show ideas. Once I learned more about the planetarium, I was pretty interested too -- 40 foot dome, Zeiss stellar projector, eight panorama projectors, a flying spot mirror... If I wanted to make some big images, this was the place! I talked my department heads into approving three independent studies, which coincidentally carried the remaining credits I needed to graduate. And I earned every single credit. I had the idea that instead of being just a dry recitation of "star facts", this would actually be a dramatic presentation. I came up with a story about a kid touring a space station. I wrote the script, directed voice actors and did over five hundred glass slides worth of 256-colour digital art (mostly drawn with a mouse). The friend who'd gotten me interested, for his sins, supplied a dynamite soundtrack, audio processing work and one of the voice actors (his son, for the main character). The planetarium staff were very helpful and supportive -- even when they had to tell me to include more star facts in the script -- and we put together a nice show. There were other tribulations -- a lightning strike the night before I was due to show it to my professors took out half the projectors! We had to scramble to source RS232 driver chips to get most of them working before the show. The show itself went off without a hitch. I was thrilled I managed to complete it. And it was big... Way bigger than the biggest television, more immersive than a flat movie theatre screen. I figured that would be the closest I ever got to IMAX. Just a few years later, I found myself working on a 3D IMAX biographical film about two Las Vegas magicians. I suffered a lot of sleepless nights, but I learned a lot on that project. I got temporarily located to Santa Monica in California to be close to the production and got to build out the network almost from scratch. I also did a bunch of rotoscoping and compositing, so I was dealing with the visual side too. We used Houdini for the 3D. Nothing Real's SHAKE and Silicon Grail's Chalice image compositing programs -- both later swallowed by Apple in one gulp. RIP. We needed to preview the 3D imagery, but this was well before 3D TV, or even useful LCD panels. We made movie files of the left and right eye images arranged horizontally, with the right eye mirrored, then would hold heavy front surface optical mirrors up to our 27 inch glass tube displays. If we wanted to check colours, we had to send in "wedge" tests of the same frame with different colour densities, then wait until it was developed to see how it would actually look. We badly needed render resources, so I negotiated with a startup that was pushing an early sort of accelerator graphics box (sort of a non-real time GPU) to loan us six of their high end machines so we could make our deadlines. Unfortunately, the accelerator didn't understand how to do alpha channels, so we had to render each layer twice if we wanted to composite it (once for RGB, once more for alpha, or opacity, information). The license for the render software ran out before we could make final delivery, so the vendor actually authorized me to hack it to extend the time so they wouldn't have to compile a new one for us. After it was in the can, I had the chance to meet one of the magicians from the film. He asked what I had worked on and I told him about the VFx. He smiled and said "Ah! You do the REAL magic!" I went on to work on more 3D IMAX films, movies (flat and 3D), television shows and short subjects, but that one "magic" moment has always stayed with me. Comments