I’ve been looking around for a scripting language that:
- has a cli interpreter
- is a “general purpose” language (yes, awk is touring complete but no way I’m using that except for manipulating text)
- allows to write in a functional style (ie. it has functions like map, fold, etc and allows to pass functions around as arguments)
- has a small disk footprint
- has decent documentation (doesn’t need to be great: I can figure out most things, but I don’t want to have to look at the interpter source code to do so)
- has a simple/straightforward setup (ideally, it should be a single executable that I can just copy to a remote system, use to run a script and then delete)
Do you know of something that would fit the bill?
Here’s a use case (the one I run into today, but this is a recurring thing for me).
For my homelab I need (well, want) to generate a luhn mod n check digit (it’s for my provisioning scripts to generate synchting device ids from their certificates).
I couldn’t find ready-made utilities for this and I might actually need might a variation of the “official” algorithm (IIUC syncthing had a bug in their initial implementation and decided to run with it).
I don’t have python (or even bash) available in all my systems, and so my goto language for script is usually sh (yes, posix sh), which in all honestly is quite frustrating for manipulating data.
The smallest footprint for an actual scripting probably will be posix sh - since you already have it ready.
A slightly bigger footprint would be Python or Lua.
If you can drop your requirement for actual scripting and are willing to add a compile step, Go and it’s ecosystem is pretty dang powerful and it’s really easy to learn for small automation tasks.
Personally, with the requirement of not adding too much space for runtimes, I’d write it in go. You don’t need a runtime, you can compile it to a really small zero dependency lib and you have clean and readable code that you can extend, test and maintain easily.
Why aren’t python and bash be available in all your systems? Which languages would be?
I would’ve recommended python, otherwise perl or Haskell (maybe Haskell’s too big) or something, but now I’m worried that whatever reason makes python undoable also makes perl etc. undoable
Mate, I came on here to post Haskell as a semi-ironic ‘joke’ and it’s included in the top comment. You’ve made my day.
I mean, it’s a functional programming language and an incredibly good one. But it probably has a far too big footprint for their use-case…
Oh, I know. I adore Haskell.
Why aren’t python and bash be available in all your systems?
Among others, I run stuff on alpine and openwrt.
I don’t need to run these scripts everywhere (strictly speaking, I don’t need the homlab at all), but I was wondering if there’s something that I can adopt as a default goto solution without having to worry about how each system is packaged/configured.
As for python, I doubt the full version would fit in my router plus as said I don’t want to deal with libraries/virtualenvs/… and (in the future) with which distro comes with python3 vs pyton4 (2 vs 3 was enough). Openwrt does have smaller python packages, but then I would be using different implementations on different systems: again something I’d rather not deal with.
As for perl, it would be small enough, but I find it a bit archaic/esoteric (prejudice, I know), plus again I don’t want to deal with how every distro decides to package the different things (eg. openwrt has some 40+ packages for perl - if I were doing serious development that would be ok, but I don’t want to worry about that for just some scripts).
You’ve defined yourself into an impossible bind: you want something extremely portable, universal but with a small disk imprint, and you want it to be general purpose and versatile.
The problem is that to be universal and general purpose, you need a lot of libraries to interact with whatever type of systems you might have it on (and the peculiarities of each), and you need libraries that do whatever type of interactions with those systems that you specify.
E.g. under-the-hood, python’s
open("<filename>", 'r')is a systemcall to the kernel. But is that Linux? BSD? Windows NT? Android? Mach?What if you want your script to run a CLI command in a subshell? Should it call “cmd”? or “sh”? or “powershell”? Okay, okay, now all you need it to do is show the contents of a file… But is the command “cat” or “type” or “Get-FileContents”?
Or maybe you want to do more than simple read/write to files and string operations. Want to have graphics? That’s a library. Want serialization for data? That’s a library. Want to read from spreadsheets? That’s a library. Want to parse XML? That’s a library.
So you’re looking at a single binary that’s several GBs in size, either as a standalone or a self-extracting installer.
Okay, maybe you’ll only ever need a small subset of libraries (basic arithmetic, string manipulation, and file ops, all on standard glibc gnu systems ofc), so it’s not really “general purpose” anymore. So you find one that’s small, but it doesn’t completely fit your use case (for example, it can’t parse uci config files); you find another that does what you need it to, but also way too much and has a huge footprint; you find that perfect medium and it has a small, niche userbase… so the documentation is meager and it’s not easy to learn.
At this point you realize that any language that’s both easy to learn and powerful enough to manage all instances of some vague notion of “computer” will necessarily evolve to being general purpose. And being general purpose requires dependencies. And dependencies reduce portability.
At this point your options are: make your own language and interpreter that does exactly what you want and nothing more (so all the dependencies can be compiled in), or decide which criteria you are willing to compromise on.
if there’s something that I can adopt as a default goto solution without having to worry about how each system is packaged/configured.
Go is probably your best bet. Simple to use, and you can compile it so it runs everywhere
I found installing Go-sdk a total PiTA. It is okay as a developer environment. But bash + gnu utils + core utils seem much more sane to me.
Of course I mostly work with Linux systems and hardly ever have to deal with Scripting for Windoze.
Sounds like you want MicroPython. It’s definitely available on OpenWrt and AlpineLinux and has a very small footprint.
If you don’t like Python, have a look at Lua/luajit.
Is compiling scripts an option? Aka compiling them in C, C++, Rust, whatever for your router on another machine, and copying and executing those binaries on your router?
Maybe something like Elvish or Nushell could be worth a look. They have a lot of similarities to classic shells like bash, but an improved syntax and more powerful features. Basically something in between bash and Python. Not sure about disk footprint or general availability/portability though
JavaScript through Node.js, or TypeScript through Deno if you like typed languages. They both check all your boxes (just check the size of the executables to make sure that it’s what you would consider “small footprint”).
Both languages and runtimes are quite popular, so you will find any answers on StackOverflow.
They are both single-executable with little dependencies, and Deno can also compile your scripts to self-contained executables.
As a bonus, both support the vast and extensive NPM package repository where you can find all sort of libraries for even the most complex tasks.
And they work with your favourite IDE or editor, not just syntax highlighting, but also contextual suggestions.
Installing node uses some 60MB (according to zypper on my current desktop). I’d rather have something small and possibly that consists of a single executable.
As a bonus, both support the vast and extensive NPM package repository
That’s not necessarily a feature :) Package repos are great if you are a developer (I am one) working primarily with that language, but are frustrating if you just want to run things.
As a Java developer, and someone who never learned Python or other scripting languages, Node is my go-to scripting language. I’ve only come around to it for that in the past year or two. But it’s great.
A scripting language written in Rust would certainly fulfill you requirement of only needing to copy one file since they are always statically linked and you can even statically compile against musl so it will work on any Linux system without needing a correct libc. Maybe check out rhai.
Tried bash, Make, and awk/sed. All hit brick walls. Finally landed on pyinvoke. Two dependencies to install on any new machine. Never had problems. Also, easy to debug and modify as projects evolve.
I use just for that usecase - highly recommended
posix sh + awk for manipulating data?
nim is great, but it is >200mb (plus AFAIK it is compiled… does it also have an interpreter?)
The part where it’s compiled is what makes it have no dependencies to actually execute
No one has mentioned PHP yet? Man, times really have changed.
I used it for scripting a decade after everyone else, but even I have to admit PHP is rarely the best choice now.
For data processing?
My go to for most of what you mention is Go, but that’s obviously a compiled language and not for scripting. Or is it - What do you think about https://github.com/traefik/yaegi, which provides an interpreter and REPL for Go? It would let you use a performant and well documented language in a more portable scripting way, but not preclude you from generating statically linked binaries if and when that’s convenient.
Realistically whatever problems you see in python will be there for any other language. Python is the most ubiquitously available thing after bash for a reason.
Also you mentioned provisioning scripts, is that Ansible? If so python is already there, if you mean really just bash scripts I can tell you that does not scale well. Also if you already have some scriptsz what language are they on? Why not write the function there?
Also you’re running syncthing on these machines, I don’t think python is larger than that (but I might be wrong).
Could use a hipster shell like fish, nushell or elvish. I know the latter two have the functional support you’re looking for.
Python.
Just remember to use pyenv for interpreter installation, version and environment management. It’s pretty straightforward that way and you have predictability.
Don’t ever manually fiddle with the system python and/or libraries or you’ll break your system. You should just rely on the package manager for that.
As for languages intended to be run as a script first and foremost, consider powershell. It’s object oriented (and not, like other script languages, just serialising/deserialising everything to JSON under the hood). The syntax takes some time getting used to, but it’s cross-platform, quite powerful, and has very good editor support.
Something I haven’t seen here yet, but may be worth considering: several programming languages support invoking the compiler to run a source file rather than compile it into a binary.
Go may be worth trying as a Python replacement. It’s strictly a programming language, but the language has been written to make it fast to compile. For simple scripts, there’s not much difference between the startup time for Python and a compiler invocation of Go.
If you want a more functional programming, Kotlinscript may be to your liking (though that comes with a rather large JVM+compiler dependency that’s not very portable). Kotlinscript is basically Kotlin (the programming language) executed like a script by putting the Kotlin compiler in the hashbang.
Similarly, Java can these days also be executed like a script if you invoke it as
java some-script.java. Using Java like this doesn’t allow for importing dependencies, though, which you may want if you ever need to process JSON.These compilers aren’t single binaries, but they are available as normal OS packages on most modern distros.
If you want to optimisme for “I want to copy a statically linked file to an alpine container”, maybe look into fish or zsh.
For some types of data manipulation, PHP may be a good fit. It’s not just for web development, although many of its more optimised features are designed for that. Useful features include clear iterations over maps/dictionaries, quite strong static types it you bother to put them in the script, and lots of examples of how to accomplish something online.
Typescript can also be invoked directly these days, and it’s even better than PHP (and arguably most programming languages) when it comes to types. I find the time it takes for the JS VM to initialise frustrating, but maybe more modern interpreters such as deno do better in this regard.
Kotlin script is fantastic! I wish it would become more popular. Dependency support, cached complier output, etc. I really like it for non-trivial scripting since you didn’t need a venv for dependencies.
OP is being ridiculous about space requirements. 60MB is a rounding error these days.
Only 5 years ago, everybody would be singing and shouting “perl”.
Nowadays it is python that has taken this position (even though Perl is still there and can do so much more).
Why would you say that Perl “can do so much more” than Python? That assertion sounds indefensible.
LOL It is one of the most well known things about perl that the language is as mighty as probably no other programming language.
Right, so you’ve got nothing to back it up. Sure, 1990s, let’s get you back to bed.
More like
20 years ago - perl
10 years ago - python
Nowadays - go
Go isn’t a scripting language
Go isn’t a scripting language, and it isn’t a system language either, despite what Wikipedia currently says. To be a system language, a language should support assembly language and shouldn’t require an embedded garbage collector. And if you’re going to make a compiled language anyway, why not make it capable of system work? Go is a platypus that’s popular with devops for some reason—probably Google’s clout in the industry.
The one thing Go does have going for is performance. It compiles and runs pretty fast. It isn’t as fast as C but it is very close.
Perl is a pain to work with
Python is a lot less pain
Even after using CPAN ? I found Perl to be much more “manageable” than Python.
Python with Venv and Pip at least works as expected. Which makes it easier.
