Lambda the Ultimate

While I haven't actually tried it, from a quick perusal of their website it seems like the goals of Scsh are to provide the kind of FP scripting language you are looking for.

Scsh is nice. I first encountered it a few years ago. It's built upon Scheme48 which is VM (image) based. On today's machines it probably starts pretty quickly, but it sort of feels like firing up Emacs versus Vi. (vi = single smallish binary; emacs = large editing environment with lots of support files).

I want to do something like:

cat foo | awk -f parseit.awk | scheme -f complexstuff.scm | awk -f present_it.awk

without a lot of disk grinding or paging!

Is there a small fully R5RS scheme that installs just as a single binary (no libs, no image, no startup files)?

Is there a small fully R5RS scheme that installs just as a single binary (no libs, no image, no startup files)?

The problem with "no libs" is that unless you build a custom binary which contains the libraries you need, you're going to end up missing lots of useful functionality. However, it ought to be possible to build a custom interpreter binary with Chicken that includes the libraries you want. Bigloo and Gambit are also possibilities.

For that kind of work, I usually use MzScheme, but I just make sure the libraries I need are installed. I don't see a problem with that - that's how Python and Perl work too, after all.

Gauche is another implementation with somewhat of a Unix scripting orientation.

I want a stripped down FP or logic language because I want Unix to supply all of the unixisms (and posixisms). Part of the brilliance of awk is that it DOESN'T do a lot outside of string processing. It can't do directory listing, file globbing and other things already provided by other unix commands.

I've got a need to do a bit of natural language parsing. I've got a pipeline of unix commands that deals with email. It downloads it from a pop3 server, splits out the mime parts, extrapolates commands from some embedded text and responds accordingly. Figuring out the commands (part of the subject line and possibly text/plain body) requires a bit of natural language processing. I'd rather do that in a programming language that is better fit for that task.

So, I have something like:

pop3download | extract_headers | split_mime | tokenize_text | figure_out_request | generate_response

I want to do the "figure_out_request" part in something like Scheme or Prolog where I can take advantage of its richer symbol manipulator capabilities. The libraries I will need (If I do need them) will be domain (NLP) specific...

An out of the box compiled chicken interpreter on cygwin yields faster startup time (using statically linked chicken binary) than perl. Nice. Very nice. It's actually on par with gawk. I haven't done anything interesting with it yet, but I am excited about its potential. (Maybe I can let gawk go back to what awk is good at and use chicken scheme for manipulating my tokenized text!)

Thanks for suggesting Chickenm, Anton

Read this if you can handle it!

is that really a parody?!

I think that he calls it a parody because it doesn't look easy to work with..

A language which replace "echo hello" by "(run (echo hello))" is on a slippery slope of being unreadable as the rest of the article quite clearly demonstrated: scsh will probably only used by people who are fans of functionnal programming, the other will keep using the classic Unix tools.

Except that you would really use (display "hello").

The case for scsh is not to replace the interactive shell. However, for writing more complex scripts than you would typically type at the command prompt, there are definitely benefits to scsh in terms of power & maintainability, compared to shell scripts.

An article like the one Luke linked has the disadvantage of having to teach most readers an entirely unfamiliar syntactic & programming paradigm, plus a powerful library for interacting with Unix, all at the same time. That's what leads to the "brutality".

there are definitely benefits to scsh in terms of power & maintainability
s/scsh/Java/ could be argued just as well, but in both cases the actual code I've seen suggests the opposite. For example, the shell code in the article is much nicer than the Scheme. The few scsh scripts that I have written (like untar) look like abominations of verbosity in hindsight (as do my Java programs).

There is an element of (CDR (ASSQ KEY ALIST)) in this, though.

P.S. I would still like to try scsh as a high-level unix system programming language instead of C.

I'm curious how you think the code could be improved. I don't know Scsh very well, but in PLT Scheme there are some libraries that would improve the code. Specifically, I'd use pattern matching more, and I'd use some utilities in the file.ss module in Mzlib (such a file-name-from-path and make-temporary-file).

shell scripting mean quick and dirty tasks execution. Most are one-liner pipes written in the shell prompt. I don't think, for instance, that Lisp's parenthesised syntax helps a lot here, specially as most command-line editors don't support parenthesis matching. And certainly one more level of indirection -- like in (run (cat "foo.txt")) -- will never be any popular for quick scripting...

There's a saying, "Use the right tool for the job".

Of course, more elaborate scripts would probably benefit from hygienic-macros, lambda abstractions, lexical scoping and elaborate list processing... the kind of scripting that is generally a messy plain block of repetitive loops and conditionals generated by automated tools. :)

I just ported my untar program from scsh to bash. I don't like the original but it is six years old so now it seems unfair to compare the results. I post this script anyway (it's pretty handy).

#!/bin/bash
# Unpack a tarfile and make sure that it ends up in a single directory
# with a name derived from the tarball (create this if necessary).
# Example: "untar foo-1.0.tar.gz" will always put the files in foo-1.0/
# Written by Luke Gorrie <[email protected]> in February 2006.

#
# Setup
#

file=$1
basefile=$(basename $file)
# we will make sure everything goes into this directory
wantdir=$(echo $basefile | sed 's/\.tar\.gz\|\.tgz\|\.tar.bz2//')
tmpdir=/tmp/untar.$$

# check args
if [ $# != 1 ]; then
    echo "Usage: untar "
    exit 1
elif [ ! -r "$file" ]; then
    echo "abort: file does not exist or not readable"
    exit 1
elif [ -d "$wantdir" ]; then
    echo "abort: $wantdir already exists" >&2
    exit 1
fi

# detect compression scheme
if echo $basefile | grep -q '\.tar\.gz\|\.tgz'; then
    compression=z
elif echo $basefile | grep -q '\.tar\.bz2'; then
    compression=j
else
    echo "Unrecognised file format" >&2
    exit 1
fi

#
# Extract & move & cleanup
#

trap '[ -d "$tmpdir" ] && rm -rf "$tmpdir" ]' SIGINT
mkdir "$tmpdir"
tar Cxf${compression} "$tmpdir" "$file"

if [ $? != 0 ]; then
    # tar failed
    rm -rf "$tmpdir"
    exit 1;
elif [ "$(ls -1 $tmpdir)" == "$wantdir" ]; then
    # The archive unpacked the way we want
    mv "$tmpdir/$wantdir" .
    rmdir "$tmpdir"
else
    # "Messy" unpack. Put it under the desired directory.
    echo "untar: creating $wantdir"
    mv "$tmpdir" "$wantdir"
fi

I can see by your example how much more cumbersome scheme scripting can be... a lot of verbosity in that scsh version.

Okay, I couldn't resist. I've taken your bash untar and compressed it further. My obfuscation of your script is here.

I've grown use to writing dense shell scripts. I find that brevity makes shell scripts more readable (hint: I compressed the awkward shell syntax for if-then-else as well as used some bash-isms to drop dependence on other sed/basename/etc).

Oh, and... neat tool Luke!

I'm suitably impressed :-)

All this talk amongst the Philistines about "cumbersome scheme scripting" is getting to me. I can't help thinking that Luke has just been dying for someone to convince him that he's wrong about s-expression based scripting. So here's my attempt to do that, for MzScheme:

#!/usr/bin/scheme-sh

(define (main args)
  (match args
    ((list _ file)
     (match-let* ((basefile (file-name-from-path file))
                  ((list wantdir ext) (split-filename basefile)))

       (or (file-exists? file) (die "abort: file does not exist"))
       (if (-d wantdir) (die "abort: ~a already exists" wantdir))

       (let ((compression
              (case (string->symbol ext)
                ((tar)        "")
                ((tar.gz tgz) "z")
                ((tar.bz2)    "j")
                (else (die "Unrecognized file format"))))
             (tmpdir (or (make-temporary-dir "untar.~a")
                         (die "Can't mkdir ~a" tmpdir))))
         (try-pk (exn:break?
                  (if (-d tmpdir)
                      (run (rm -rf ,tmpdir))))
           (|| (tar ,($ 'Cxf compression) ,tmpdir ,file)
               ((rm -rf ,tmpdir) (exit 1)))
           (cond
             ((equal? (run (ls -1 ,tmpdir)) (list wantdir))
              (&& (mv ,($ tmpdir '/ wantdir) ".") (rmdir ,tmpdir)))
             (else 
               (printf "untar: creating ~a\n" wantdir)
               (run (mv ,tmpdir ,wantdir))))))))
    (else (die "Usage: untar filename(.tar|.tar.gz|.tgz|.tar.bz2)"))))

This relies on some generic support routines, like run, to do some scsh-like things. A version complete with comments, syntax highlighting, and all the necessary support code can be found at ProcessUntarScriptExample in the Scheme Cookbook. The scsh-like features are documented (briefly!) in the source code for the support routines.

This code is more likely to win an award for obfuscation than readability, but it's not too different from the "compressed" bash version in that respect. I wanted to keep it as close to the bash version as practical, while still using Scheme features instead of shell features for everything except actually executing external programs. I even went so far as to (define -d directory-exists?), more for fun than anything else.

I even went so far as to (define -d directory-exists?), more for fun than anything else.

Yes, that was fun...

I can't help thinking that Luke has just been dying for someone to convince him that he's wrong about s-expression based scripting.

Stranger things have certainly happened on LtU :-)

I like your example. If it were a stand-alone scsh program then I would concede the point, but really I have to count the several-page ad-hoc scsh implementation towards verbosity :-)

Notes: I don't like the need for string->symbol, that seems very low-level to me. I also just noticed a bug that stops these programs from working on the foo-1.0.tar.gz filename in the comment (just mentioning).

I know it should prove to be pretty cool if we had mkast, asted and ast2src, little utilities to do arbitrary AST processing.

Like:
$ mkast source.c | asted -e 's/foo/bar/gi' | ast2src -pp k&r > source2.c

makes an AST from the source code, substitutes (g)lobal (i)dentifier "foo" for "bar" and converts the ast back to source code using k&r pretty printer... yes, it would leave local identifiers "foo" alone, just change the references for the global one.

or
$ mkast source2.c | asted -e 's/(Person).surname/$1.last_name/' | ast2src -pp k&r > source3.c

This should change the definition of the field "surname' from (.) struct "Person" to be "last_name" and accordingly to propagate this change throughout all the AST, in all references for surname. Just a mere text substitution like sed does wouldn't cut it.

of course, by now you realize asted acts like sed. ;)

how about it? Gosh, i would kill for such tools and vim and emacs integration... :)

I know it should prove to be pretty cool if we had mkast, asted and ast2src, little utilities to do arbitrary AST processing.

Unfortunately, this is where the text-based nature of the Unix pipeline starts to stretch at the seams a little. Since in order to be "well-behaved" from the traditional Unix point of view each of these tools ought to accept and emit plain text, we'd need a standard serialization format for these ASTs. Of course, such a format already exists: the C programming language. Establishing a more machine-friendly interchange format would be pretty easy, but it comes at the expense of interoperability. And since interoperability is really the whole point of the pipeline approach, that's a pretty heavy price...

This is one reason why this kind of tool typically becomes pretty monolithic in Unix. Attempts to allow typed data pipelines (as in Microsoft's new shell) are a great idea, but bringing this to Unix would probably require establishing a new data processing hegemony and re-implementing ls, ps and their many friends, which of course flies in the face of Unix culture in many respects.

I'd really like to use a shell with these capabilities, though, and as Microsoft will probably push us in this direction, I think we'll see more of this kind of thing.

By the way, for a step in this direction, you might be interested in XMLStarlet.

"Establishing a more machine-friendly interchange format would be pretty easy, but it comes at the expense of interoperability. And since interoperability is really the whole point of the pipeline approach, that's a pretty heavy price..."

I don't think the price would be that high as long as the interchange format remains textual. It doesn't even have to be structured xml, though i believe a W3C-sanctioned xml format for ASTs would greatly speed up adoption. It certainly shouldn't be C code in this day and age.

"bringing this to Unix would probably require establishing a new data processing hegemony and re-implementing ls, ps and their many friends, which of course flies in the face of Unix culture in many respects."

as long as its text, there should be no problem. It should be no problem if only mkast, asted and ast2src were the only utilities to truly understand the underlying format. sed or awk do not understand what is passed to them, they just do simple textual substitutions. If there are no tools for general-purpose structured processing, i think it's easier to just add new ones than making the old behave likewise.

"By the way, for a step in this direction, you might be interested in XMLStarlet."

Yes, that's a nice start as far as xml processing from the command-line goes. Now we just need a standard xml format for ASTs... the java guys have such one in one of Eclipses plugins, but they don't mind command-line processing anyway... :/

I just curious here, what would you like to use the light-weight constraint solver for? I can't really imaginge when one would use such a thing.

Lightweight may be too strong of a word ;-) I am thinking of a very specific application that involves scheduling/rescheduling around a meeting calendar (where the primary interface is email -- but that's a different topic!)

As part of a unix pipeline workflow, I would like to be able to feed the solver meeting requests that may conflict with an already populated calendar and have it propose scheduling solutions.

I admit that this is a rather specific need ;-)

I probably should have said a I was hoping for a "lightweight" programming/scripting language wrapped around a constraint solver library. I don't want the weight (from a Unix utility perspective) of an Alice or Oz. I haven't been keeping up with Gecode, so I don't know if it has been bound to any "scripting capable" language yet.

A potentially really bad analogy would be that I want a troff, not a TeX.

Ok, now I get the picture.

What one would want in your application is good abstractions for scheduling requests, so a model could be written cleanly. On the other hand, it sounds like you have quite a static problem (although the instance-date is not), and then a C++ program would do the trick, I believe.

There is no lightweight wrapper for Gecode (yet), Java being the only language other than C++ that we are currently looking at.

What do you mean by "lightweight"? I usually see that word to describe incomplete projects that don't fully implement their specifications. i.e. projects that strongly favour NJ-style (aka 'worse is better').

Searching through Freshmeat provides a lot of examples.

I mean something that is not a world unto itself. It should have a smallish footprint and very quick startup time (as it may be invoked many times as part of a pipeline). It doesn't need networking capabilities, regular expression parsing, etc because there are other tools available that do that well.

By no means do I want incomplete!

Regarding "worse is better". I am starting to appreciate the NJ style more these days. In particular, I am more and more impressed with what NJ (Bell labs) did themselves, not the often "pale" attempts at that style by others. Bourne shell, C, Awk, troff, etc were very high quality tools that were written to fill specific needs. They were fed by a healthy dose of research with a bias toward practicality. (Plan 9 continued that tradition). I miss the quality and depth of thought that went into these Bell lab tools in a lot of the stuff produced today. Today we have a lot of bloated software that wants to do everything.

I've been scouring the net looking for a lightweight Scheme (something that doesn't feel the need to include the kitchen sink because the kitchen sink -- under Unix -- already exists!).

Chicken Scheme (the interpreter) looks really, really interesting to me because I can jettison a lot of additional stuff and add in what I need. It starts up really fast and it appears that if all I need is the interpreter binary (along with my Scheme scripts), then that is all I need to deploy.

But, what use is a lightweight Scheme (something that just implements R5RS)? Well, I've been happily passing flat streams of data through my unix pipeline with Awk and other tools. But, now I have a need to manipulate more complex data structures and relations. The end result is once again a flat stream of data, but I prefer to use something good at manipulating nested lists and trees... sounds lispy, eh?

Hear, hear. Maybe blame XML - but even before that, if one were dealing with records in files I think the line-by-line nature of the unix utilities really limited what could be bung'd together. That's why Perl met with such success; it lifted what people already knew from shell programming into a more stateful world that could handle "ok, I'm in a record now, do stuff once the record is fully read in" that wasn't doable with piped things on the command line.

It looks like one needs to have a haskell to bootstrap it, so I don't have any idea how well it works, but this at least sounds interesting:

h4sh provides a set of Haskell List functions as normal unix shell commands. This allows us to use Haskell in shell scripts transparently.