System X Shell

When you enter command sxx without arguments in the Windows Terminal window (wt.exe), the sxx virtual machine starts a System X shell session.

The command sxx sh --debug starts the shell in debug mode. When in debug mode, the token expansion information and command line parsing results are printed to the terminal along with the command output.

When you type a command line in the shell and press ENTER, the shell

chops the command line into tokens
expands the tokens
parses the tokens into commands
executes the commands

Tokenization

First the command line is chopped into tokens consisting of meta-characters and series of other characters classified as tokens. The following table lists the tokenization rules:

rule name		rule content	token name
sep	→	space \| tab	SEPARATOR
space	→	' '	SPACE
tab	→	'\t'	TAB
quoted-char	→	'\\' any	QUOTED_CHAR
single-quoted	→	\'[^']*\'	SINGLE_QUOTED
double-quoted	→	\"[^"]*\"	DOUBLE_QUOTED
and-if	→	&&	AND_IF
or-if	→	\|\|	OR_IF
dsemi	→	;;	DSEMI
dlessdash	→	<<-	DLESSDASH
dless	→	<<	DLESS
dgreat	→	>>	DGREAT
lbrace	→	{	LBRACE
rbrace	→	}	RBRACE
bang	→	!	BANG
pipe	→	\|	PIPE
lparen	→	(	LPAREN
rparen	→	)	RPAREN
less-and	→	<&	LESSAND
great-and	→	>&	GREATAND
less-great	→	<>	LESSGREAT
clobber	→	>\|	CLOBBER
less	→	<	LESS
great	→	>	GREAT
amp	→	&	AMP
semi	→	;	SEMI
back-quote	→	`	BACKQUOTE
token	→	[^ \t\\'\"&\|;<>{}`]+	TOKEN
word	→	token	WORD

The backslash character quotes the character coming after it. A token enclosed in apostrophes quotes the enclosed characters. A token enclosed in double quotes the enclosed characters. Tokens containing quotation are not subject to token expansions. Exception to this is the filename expansion, in which backslash-quoted characters are included in the filename.

Token expansions

After tokenization the tokens are expanded using the following expansions:

brace expansion
tilde expansion
parameter expansion
filename expansion
quote removal

Brace expansion

In brace expansion the tokens are grouped to form brace expressions consisting of

a possibly empty preamble
comma-separated list of tokens enclosed in braces and
a possibly empty postfix.

These groups are separated by sep tokens.

The expansion is purely textual.

Example

command	prints
echo a{c,p,t}e ate all the bananas	ace ape ate all the bananas
echo file{.txt,.bak,.log}	file.txt file.bak file.log

Tilde expansion

In tilde expansion tokens starting with a tilde character ('~') are expanded as follows:

If a token contains a slash ('/'), characters after the tilde and before the slash form a tilde prefix, otherwise characters after the tilde form the tilde prefix. Thus a tilde prefix can be empty.

If the tilde prefix is empty, and the token contains a slash, the token is expanded to contain current user's home directory path followed by the characters of the token starting with the slash.

If the tilde prefix is empty, and the token does not contain a slash, the token is expanded to contain current user's home directory path.

When the tilde prefix is not empty, the tilde prefix is considered as a user name. If the token contains a slash, the token is expanded to contains that user's home directory path followed by the characters of the token starting with the slash. Otherwise the token does not contain a slash, so the token is expanded to contain that user's home directory path.

Example

command	prints
echo ~	/home/seppo
echo ~/foo	/home/seppo/foo
echo ~root	/
echo ~root/bin	/bin

Parameter expansion

When a token contains an unquoted dollar character ($), the token is subject to parameter expansion.

The parameter expansion expands a name of an environment variable to the value of that environment variable.

The name of the parameter can be optionally enclosed in braces.

Example

command	prints
echo $PATH	/bin
echo ${SHELL}	/bin/sh

Filename expansion

When a token contains a pattern character '*', '?' or '[', the token containing the pattern is expanded into path tokens containing filenames found in the path prefix before the pattern.

Example

command	prints
echo /etc/*	/etc/environment /etc/group /etc/passwd /etc/passwd_hash /etc/sudoers
echo /mnt/sx/system/System.Lex/*.cm	/mnt/sx/system/System.Lex/ClassMap.cm /mnt/sx/system/System.Lex/Keyword.cm /mnt/sx/system/System.Lex/Lexeme.cm /mnt/sx/system/System.Lex/Lexer.cm /mnt/sx/system/System.Lex/ParsingException.cm /mnt/sx/system/System.Lex/ParsingLog.cm /mnt/sx/system/System.Lex/Span.cm /mnt/sx/system/System.Lex/Token.cm /mnt/sx/system/System.Lex/TrivialLexer.cm /mnt/sx/system/System.Lex/XmlParsingLog.cm

Quote removal

The last token conversion phase is to remove quotation and to convert tokens into words.

Parsing

After the tokens are expanded, the shell parses the expanded tokens into command objects according to the following grammar:

rule name		rule content	expression
command-line	→	list separator?
list	→	and-or ( separator and-or )*	sequence
separator	→	semi
and-or	→	pipeline ( and-if pipeline \| or-if pipeline )*	and-or
pipeline	→	bang? command (pipe command )*	pipeline
command	→	simple-command \| compound-command
simple‑command	→	command-prefix (command-name command-suffix? )? \| command-name command-suffix?	simple‑command
command‑prefix	→	(io‑number? io‑file \| assignment‑word)+
command‑name	→	word
io‑number	→	word containing only digits
io‑file	→	less‑and filename \| less filename \| great‑and filename \| great filename \| dgreat filename \| less‑great filename \| clobber filename	I/O redirections
filename	→	word
assignment‑word	→	word containing an assignment symbol (=)	assignment
command‑suffix	→	(io‑number? io‑file \| word)+
compound‑command	→	brace‑group \| subshell
brace‑group	→	lbrace command-line rbrace	brace-group
subshell	→	lparen command-line rparen	subshell

Commands

Command sequence

Execute a list of commands separated by the ; control operator. The exit code of the command list is the exit code of the last executed command.

Example

command	prints
echo hot >meal; echo dog >>meal; cat meal	hot dog

And-Or command

Execute command expressions separated by the && and || operators from left to right.

&& operator

The left-command && right-command expression is executed so that the right-command is executed only if the exit code of the left-command is zero. The exit code of the command expression is the exit code of the left-command if it is nonzero, otherwise it is the exit code of the right-command.

Example

command	prints
echo foo && echo bar	foo bar

|| operator

The left-command || right-command expression is executed so that the right-command is executed only if the exit code of the left-command is nonzero. The exit code of the command expression is the exit code of the right-command if the exit code of the left-command is zero, otherwise it is the exit code of the left-command.

Example

command	prints
rm nonexistent \|\| echo NOT REMOVED	System.SystemError: path 'nonexistent' not found... NOT REMOVED

Pipeline

| operator

The shell executes the left-command | right-command expression by directing standard output of the left-command to standard input of the right-command. The shell waits the commands of the pipeline to terminate and then returns the exit code of the right-command as the exit code of the pipeline.

Example

command	prints
cat /etc/environment \| head --lines=1	PATH="/bin"

Simple command

Executes the variable assignments and I/O redirections contained by the command-prefix and the I/O redirections of the command-suffix. Then if the simple command contains a command-name executes that command with arguments specified in the command-suffix.

Example

command	prints
echo foo bar baz	foo bar baz

Assignment command

NAME=VALUE

assign VALUE to the variable NAME in the command's environment; if simple-command contains no command-name, assigns VALUE to the variable NAME in the shell's environment

Example

command	prints
FOO=BAR
echo $FOO	BAR

I/O redirection commands

<filename	open 'filename' for reading and redirect its contents to command's standard input (file descriptor 0)
N<filename	open 'filename' for reading and redirect its contents to command's file descriptor N
>filename	open 'filename' with create, truncate, and write flags and redirect commmand's standard output (file descriptor 1) to it
N>filename	open 'filename' with create, truncate, and write flags and redirect command's file descriptor N to it
>>filename	open 'filename' for appending and redirect command's standard output (file descriptor 1) to it
N>>filename	open 'filename' for appending and redirect command's file descriptor N to it

Example

command	prints
echo hot >meal
echo dog >>meal
cat meal	hot dog

Brace group

Executes the commands contained by the brace group in the current shell process.

Subshell

Executes the commands contained by the subshell expression in the subshell child process of the current shell process.

Internal commands

The following table contains the commands built into the shell:

command	description	notes
cd [ARG]	changes current directory	if the cd command is invoked with a directory argument, changes the current directory to that directory, otherwise if the command is invoked without arguments, changes the current directory to the home directory of the current user.
exit	exits the shell
printenv	prints the environment
unset ARG	removes an environment variable from the shell's environment
umask [THREE_DIGIT_OCTAL_NUMBER]	sets or prints the umask	if the umask command is invoked with an argument, changes the current umask to THREE_DIGIT_OCTAL_NUMBER for owner, group and others and prints the value of previous umask, otherwise if the command is invoked without arguments, prints the current umask.