Sunday, December 7, 2014

Process Invocation

Process invocation is important. There can be a portable abstraction and OS-specific implementations that Igor (and in particular tasks) can turn to. In the following I will call it a PIU (Process Invocation Unit) and its ancillary job is to mediate data between processes.

The workbench project already requires a new generation of tools that need to be more prudent about where and how to store user visible data in a multi-device single-user one-world. So why not use this passage to revisit process invocation.

Aspects of a Process

I find it very revealing to use the Six W questions to find out what all the factors of process invocation are.

What: The code that is going to be executed. You will sometimes encounter a hierarchy of subcommands that eventually leads to exactly one operation or application to execute. Each tier in this hierarchical grouping might accept individual configuration options. My impression is that, beyond sharing functionality, these groupings often pertain to a conceptual workflow and/or share the same application-private data.

Who: The data to operate on. This usually involves product-data and not application-private data, which the application intrinsically knows how to find.

How: The way in which the operation is performed; the configuration.

When: The time or the event that triggers the operation.

Where: The physical location, as well as the system and the WS-node, where the process will be launched. In the context of the green/red node distinction I usually called this part the privacy level (public, drone, private). There is also the WB and module location. This is in some way special as it indirectly specifies the data (who) to work with.

Why: Why not? This can be captured in comments and ultimately depends on your individual meaning of life.

From these 6 categories, the who and how are information that is of interest to the process itself, whereas the other have a propensity of being managed externally. You can, of course, imagine a process that reschedules itself or iterates several locations, but for the predominant use case what/when/where/why would be handled by an external facility (or the user).

(Who is what and what is who? It is a little bit ambiguous, but the question of what code to execute seems to be more appropriate.)

Language Requirements

Let's turn the focus to the places where process invocation happens. Apart from starting one particular application (for example through a desktop) there are command-line and scripting languages at the process invocation (PI) level. I think these two contexts have diametrically different requirements and need to be strictly kept separate.

In a script you aim for understandability, use explicit expressions, and are able to comprehensively structure the script. Most importantly, you write code once and read it many times.

In contrast, the command-line is supposed to allow you to formulate jobs in a very concise way. You will often try to combine a series of operations on one line, to be able to later refine it after accessing it in the history. In this sense you might read it back once or twice, but only within a very short time frame.

A concrete example: In a scripting language with text objects (which may be endpoints of streams) you can add a statement to write content out to a file on a separate line, whereas on the command-line you would want something like the redirection features that exist today. The same goes for pipes. In a modern PI-scripting language you would be able to easily capture an output stream in a text object and stream it to another process on the next line. Incidentally, having text-objects on the command-line and a way to inspect them in tabs might make the difference less stringent. You may also be able to further filter such a stream with methods on the text-object and inspect the result in real-time.

In addition, note that a scripting language at the PI-level has also quiet different requirements from (what I frivolously call) an in-process scripting language (eg Python, Perl). A PI-language needs strong orchestration features.

Text-objects would take on a special role, as text is the common ground to be able to combine processes implemented in different languages or even incorporate remote execution. There are worldwide standardization efforts to represent textual data.

Process Input

A process needs to know how to operate on who. Let's look at each more closely.

You usually would not pass data as arguments, simply for the unwanted parsing that might happen under some conditions. Therefore you use streams and files. Having a range of streams designated as input streams with optional labels would often avoid the need for potentially insecure temporary-file handling. From a programming perspective processes would gain more purity (referential transparency). You would need less predefined locations, in particular for small utility tasks.

Determining the effective configuration (how) is quiet tedious nowadays. In some cases, you have to parse a configuration file, check the environment, then parse the command line, and eventually merge the results. All just to obtain the relevant values for how the operation is supposed to transpire.

In the ideal case, an application provides a configuration specification, and a process obtains a structure of the applicable configuration through one call or as a special value similar to argv.

A command-line or script could streamline the arguments of a call to normalize away the syntax flavor, and pass it to Igor/PIU for translation (eg short to long keys). Subsequently the effective configuration would be produced and provided to the process.

The how parts of a command-line processed in such a way can be seen as ad-hoc OTC (One Time Configuration), whereas OTC prepared in the global configuration and activated per call would be recorded OTC.

In terms of performance, note that this unified approach includes what would nowadays be a global configuration (eg in form of a file). If you are worried about the time this might cost (despite optimized data structures), you can of course work exclusively with unprocessed ad-hoc OTC, which would replicate a simple array of strings as you know it from argv.

Process Output

Inside a process you would be able to obtain temporary streams to manage large amounts of data, and later designate them as output streams. Those can again be labeled and there can be multiple. In essence, this would allow you to create pipes of multiple streams (multi-pipes). If the PIU recognizes a legacy script (or a dedicated single stream application) it may even be capable to split and join multiple streams for that stage. Text-objects with corresponding methods might reduce the demand for such filter applications, though.

Then there is the opportunity to create structured output and a consumer would be able to specify which part of the output it is interested in.

Imagine an output stream with the special type "structured". This isn't actually a stream but a key-value string mapping (possibly with nesting and lists) stored inside the PIU. A process that is capable of accepting structured input would access that record directly. Otherwise, that mapping may have a string representation which can be streamed to standard output and the user.

In a way, this does also make the differentiation between a script and user consumer explicit. Depending on which stream is being read, the structured data output stream can be transformed into a pretty-printed standard output text automatically, or the process might create a custom-made output for the user.

A command line syntax could have special syntax for output picking and only the selected values would be presented to the user. Right now, this is sometimes accomplished by adding command-line options that influence the output composition. In conclusion, output picking is a kind of how.

In what way would these multiple output streams be presented in a terminal?

In a GUI-terminal, the actual terminal display could be reduced to user-IO and thereby unclutter the user's interactions. Data-streams could be displayed in tabs above that terminal, with the option of additional filters applied. The user would be able to create a text-object that references such a stream and thereby keep it alive. Of course, there could also be a history of output data-streams per prompt.

In a pure text terminal things would not be that convenient. Everything can certainly be redirected to standard output with a result like today's shared output. Maybe a stream selection mechanism, a "screen"-like setup, a multiple-window terminal, or configurable separators can help.

PI Language Pluralism

As mentioned above, PI scripting and the command line have different requirements. But what is even more interesting is the fact that there can actually be a whole range of languages, like they exist at a lower level.

In my opinion, the most striking omission from what's available for process orchestration scripting (at least for Unix shells) are text-objects and exceptions. I personally think that the higher you get on the language ladder, the more useful exceptions become.

If you think of text-objects as stream-ends, this will open up a whole new (and I think easier and more reliable) way to establish concurrent operations through process orchestration. Together with the option of multi-pipes, it would be possible to assemble a range of such text-objects, wait on them, and stream all to another process at once.

Mappings at the PI-level

Essentially, this would establish a mapping-like structure at the PI-level that can be nested. There would be list-values, but other than that, all keys and values would be strings.

I think we could make a lot of progress with such a relatively simple structure. It would be eligible for the effective configuration, labeled streams, and structured IO. Maybe it would even make sense to store selected input data in the configuration, or use structured output as ad-hoc OTC.

The string values could serve as a basis for other secondary types (bool, numbers, etc). The type-info would just be carried along and processes would interpreted it themselves. Note that the ability to have fragmented records and nesting reduces the need for composite text fields like URLs.

Signature and Polyglot

To recap, the effective configuration is a mapping with known keys from the configuration specification.

Notice what happened by improving the how aspect (and adding output picking). You can now easily check the signature of process invocation. You have a configuration structure that can be verified against the specification and a program can also determine what data it accepts and in what form.

This means that you can more reliably formulate tasks to be scheduled elsewhere. Igor can query the corresponding PIU for supported features and inform outright if that task is even possible.

In addition, you obtain better guarantees (and thereby trust) when combining orchestration scripting with in-process scripting. Concretely, when you assemble data from diverse processes but would like to process that information with a more specialized in-process utility script, you could get a more reliable association to said utility script and can employ static checking. Absence of the need for temporary files would make things easier, too. The location-graph can facilitate packaging (in particular keeping track of an installation). Using the right tool for the right task may become a lot easier.

Configuration is fragmented

You will not get all of the applicable configuration at program startup. Specific configuration can apply to selected files or streams. You would be able to query the configuration that is different to the base configuration for such an item.

Igor/PIU would handle merging and filtering of effective configuration at different levels in the call stack. For example, the ad-hoc OTC of a call would have to be merged with the active global configuration/OTC. Then, the configuration specification would describe what configuration is accessed by a particular process and it could be filtered accordingly.

Igor would know about configured (global) aliases and be able to resolve them before further configuration processing.

Igor/PIU would cover all your configuration needs. And it could propel #hurling-the-beacon.

Update (9, Dec, 2014)

The mapping would be more like a linked list than a hash. Ordered and sequential access. Maybe buffering can be solved by determining exactly one field per stream that can grow without boundary.

All the information in these records is meaningful to the process. The mapping simply serves as a vehicle to transmit that info. The process can choose what to do with that information and optimize access.