1. Introduction¶
spy is a Python CLI. It’s quite powerful, as you’ll see below, but let’s start with the basics: you feed it a Python expression, it spits out the result.
$ spy '3*4'
12
There’s no need to import modules—just use them and spy will make sure they’re available:
$ spy 'math.pi'
3.141592653589793
1.1. I/O¶
Standard input is exposed as a file-like object called pipe
:
$ cat test.txt
this
file
has
five
lines
$ spy 'pipe.readline()' < test.txt
this
It’s a io.TextIOBase
, with a couple of extra features: You can index
into it, or convert all of stdin into a string with str()
.
$ spy 'pipe[1]' < test.txt
file
$ spy 'pipe[1::2]' < test.txt
['file', 'five']
$ spy 'str(pipe).replace("\n", " ")' < test.txt
this file has five lines
Passing -l
(or --each-line
) to spy will iterate through
stdin instead, so your expressions will run once per line of input:
$ spy -l '"-%s-" % pipe' < test.txt
-this-
-file-
-has-
-five-
-lines-
spy helpfully removes the terminating newlines from these strings. If you don’t
want that, you can pass --raw
to get stdin
unadulterated.
$ spy -lrc repr < test.txt
'this\n'
'file\n'
'has\n'
'five\n'
'lines\n'
1.2. Piping¶
Much like the standard assortment of unix utilities, which expect to have their inputs and outputs wired up to each other in order to do useful things, each fragment processes some data then passes it on to the next one.
Data passes from left to right. Fragments can return the special constant
spy.DROP
to prevent further processing of the current datum and
continue to the next.
$ spy '3' 'pipe * 2' 'pipe * "!"'
!!!!!!
$ spy -l 'if pipe.startswith("f"): pipe = spy.DROP' < test.txt
this
has
lines
1.3. Limiting output¶
-
--start
=<integer>
,
-s
<integer>
¶ Start printing output at this zero-based index.
-
--end
=<integer>
,
-e
<integer>
¶ Stop processing at this zero-based index.
-s
and -e
mirror Python’s slice semantics, so -s 1 -e 3
will show
results 1 and 2. This means -e
on its own is equivalent to a limit on the
number of results.
Once the result specified by -e
has been hit, no more data will be
processed.
1.4. Data flow¶
Before we construct anything more complex, a brief discourse into how data moves around in spy: Each fragment in spy tries to consume data from the fragment to its left. It processes it, then yields to the fragment to its right, which will do the same thing. To run the program, spy just tries to pump as much data out of the rightmost fragment as it can—everything else is handled by the fragment mechanic.
In the examples I’ve given above, each fragment has consumed and yielded data on
a one-to-one basis, but there’s no inherent reason for that restriction.
Fragments can yield or consume (or both) multiple values using
spy.many
and spy.collect
, respectively.
1.5. Decorators¶
In one example above, we used an if
statement to filter by a predicate.
That’s far from elegant—by my rough guess, about half the characters in the
fragment are boilerplate. spy provides some function decorators to avoid
repeating this and a few other common constructs—they’re available as flags from
the CLI:
-
--accumulate
<fragment>
,
-a
<fragment>
¶ passes the the result of
spy.collect()
to the fragment.
-
--callable
<fragment>
,
-c
<fragment>
¶ calls whatever the following fragment returns, with a single argument: the input value to the fragment.
-
--filter
<fragment>
,
-f
<fragment>
¶ filters the data stream, using the fragment as a predicate: if it returns any true value, the data passes through, but if it returns a false value
spy.DROP
is returned instead.
-
--keywords
<fragment>
,
-k
<fragment>
¶ executes the fragment using its own input value as the local scope, which must be a mapping. Names from the global scope (but not
pipe
) are still available unless shadowed by keys in the input mapping.
-
--many
<fragment>
,
-m
<fragment>
¶ calls
spy.many()
with the return value of the fragment (which must be iterable).
1.5.1. Literal decorators¶
Literal decorators are a kind of decorator that accept string arguments rather than Python code.
-
--interpolate
<string>
,
-i
<string>
¶ uses
<string>
as astr.format()
format string on the input. Positional parameters like{0}
index into the input value, and named ones access the local scope of the fragment, so the full input value is available as{pipe}
.$ spy -li '-{pipe}-' < test.txt -this- -file- -has- -five- -lines-
-
--regex
<string>
,
--regexp
<string>
,
-R
<string>
¶ matches the input against
<string>
as a regexp usingre.match()
.$ spy -lR 'f.*' -fc id -i '{0}' < test.txt file five
1.6. Deferred application¶
spy overloads callable objects (when they’re builtins or autoimported) to add
implementations of most Python operators. These return a function that calls the
original function and then applies the specified operation. They take a single
argument only, and are essentially just a shortcut that lets you avoid typing
(pipe)
in some cases:
$ spy '[1,2,3]' -c 'sum/2'
3.0
$ spy '[1,2,3]' -c 'sum/len'
2.0
1.7. Doing stuff¶
Nothing here is particularly useful in isolation. Let’s throw it all together by
pretending we’re jq
:
$ spy -lc json.loads -fk '"Rutile" in export_commodities' -k name -e 10 < stations.jsonl
Hieb Orbital
Hahn Terminal
Anderson Colony
So-yeon Mines
Williamson Enterprise
Julian Hub
Fancher Enterprise
Neville Vision
Raleigh Terminal
Arrhenius Beacon
Note how -l
trivially gives us newline-delimited JSON, a job which was
previously so hard it required its own top-2000 PyPI package!
1.8. Exception handling¶
If your code raises an uncaught exception, spy will try to intercept and reformat the traceback, omitting the frames from spy’s own machinery. Special frames will be inserted where appropriate describing the fragment’s position, source code, and input data at the time the exception was raised:
$ spy 'None + 2'
Traceback (most recent call last):
Fragment 1
None + 2
input to fragment was <SpyFile stream='<stdin>'>
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
If an exception is raised in a decorator outside the call to the fragment body, the fragment is mentioned anyway. This is not strictly true, given that none of the code in the fragment takes part in the call stack in this case, but this particular lie is almost universally more useful:
$ spy -c None
Traceback (most recent call last):
Fragment 1, in decorator spy.decorators.callable
--callable 'None'
input to fragment was <SpyFile stream='<stdin>'>
TypeError: 'NoneType' object is not callable
The philosophy here is that what made it go wrong is more interesting than
exactly how it went wrong, so that’s what spy gives you by default. You can
get the real traceback by passing --no-exception-handling
to spy.