1 18-InputOutput

Previous: 17-Exceptions.html

On this page, we cover the Linux pipe, Standard I.O. redirection, and Python file handling!

1.1 Screencasts

1.2 Program-internal function input and output (I.O.)

Within a function, you can access other variables in your program in a variety of ways:
For example in python:

def procedure(input_parameter_values, thing_by_reference) -> return_value:
    global some_global
    some_global = value
    thing_by_reference[1] = value

Heap (and stack) we deal with later, in C and C++

1.3 Standard in and out (standard stream IO)

This is actually an OS/Shell topic (bigger than just python).
Command line programs use standard input, output, and error, to interact with the user.
Graphical User Interface programs generally do not use this type if IO.


From within your choice of shell,
you can trade data between processes running within the operating system,
as well as the keyboard, monitor/screen, and files on disk.
The keyboard is used to enter a command,
and respond to an interactive program/process,
and the process spawned by the command prints standout output and error to the monitor/screen:

After a program/process is run,
input from keyboard can be interactively sent to the process,
and output from process sent to the monitor:

Two output streams exist,
one for general output,
the other for error output:
While these both head to the screen for your viewing,
a process can write to files as well (more to come later).

The streams are numbered pseudo-file devices in Linux:
Programs/processes can interact with these pseudo-file input/output devices.

A command line utility also has an exit code,
which communicates it’s success (by convention):

There are many inputs and outputs for a process within an operating system:

+++++++++++ Cahoot-18a-1

1.3.1 Redirection

Standard IO manipulation via shell redirection using the special characters:
< and >

One can intercept standard output before it hits the default monitor,
and instead redirect it to a file:
For example


ls >file.txt
cat file.txt
less file.txt

# This overwrites the above file
echo hey >file.txt
less file.txt

# This appends to the above file
ls >>file.txt
less file.txt

# This produces error
ls nonexistentfile

# Re-directing std-err
ls nonexistentfile 2>stderr.txt
less stderr.txt

# What shows up here, when file.txt exists?
ls nonexistentfile file.txt 2>stderr_only.txt
less stderr_only.txt

# Send both std-err and std-out to a file:
ls nonexistentfile file.txt >stderr_and_stdout.txt 2>&1
less stderr_and_stdout.txt

# Alternative, to do the same:
ls nonexistentfile file.txt &>stderr_and_stdout.txt
less stderr_and_stdout.txt

One can trick your program into thinking that a file’s contents are being typed into it’s interactive standard input:
For example:


read realtyping
# type something
echo $realtyping

# Redirect hey into a file for use in capturing output below
echo hey >temp.txt

# Intercept output into temp.txt
read trickedread <temp.txt
echo $trickedread

Or with python

# -*- coding: utf-8 -*-

print(input("Type something: "))

python3 std_io_00_echo.py
# type something, watch it echo

# Capture output whiles still typing
python3 std_io_00_echo.py >capturedout.txt
# type something
less capturedout.txt

# Feed input, print to std-out
python3 std_io_00_echo.py <capturedoutput.txt
less capturedout.txt

# Feed input and capture output
python3 std_io_00_echo.py <capturedout.txt >metacapturedoutput.txt
less metacapturedoutput.txt

+++++++++++ Cahoot-18a-2

1.3.2 Pipes

One can send data from one program’s standard-out to another program’s standard-in.
The stdout/stdin arrows at right hand side are pipes:
The command process1 | process 2 intercepts stdout from the first process and feeds it into stdin for the second process:
For example:


ls | less
echo "hey there you" | grep --color=auto hey
echo "hey there you" | grep --color=auto there
ls | sort | less

# hash a string with a pipe...
echo smoke | sha256sum

These tools allow you to blaze through data on your filesystem!

Another example:


ls -l | grep '^d' | sort | less

# or a sorted list of the python files in the directory
ls | grep ".py" | sort | less

+++++++++++ Cahoot-18a-3

1.3.3 Summary of stdio, redirection, and piping


1.3.4 Python internal standard I.O.

While this is an OS/Shell topic,
there are some neat python tricks,
like overwriting sys.stdin and sys.stdout in your program.
See code for examples: 18-InputOutput/std_io_01_stdio.py

1.3.5 Subprocess std-io from within python

The subprocess module allows running various system shell commands from within python.
See code for examples:

1.3.6 Mock/patch

Some unit testing frameworks allow you to simulate standard input and output (but overwriting sys.stdin and sys.stdout is usually better/easier):

1.3.7 Un-buffered non-blocking keyboard input

What if you want python to use the characters you type in, but without hitting enter?
* See code for a pure-python example: 18-InputOutput/std_io_02_getch.py
* or use ncurses:
* https://docs.python.org/3/library/curses.html
* https://docs.python.org/3/howto/curses.html

1.3.8 Conclusions

Command line programs employ standard IO,
and GUI programs (graphical user interface) programs do not.


1.4 Direct python file I.O.

How do we read and write files in python?
This type of input can work for BOTH a command line program, and a GUI (graphical user interface) program.

1.4.1 File types

Come in a couple major forms (with lots of sub-types):
1. Text (un-structured text, structured text, etc.)
2. Binary (At least some of the file is actual byte-based data.)

1.4.2 1. Text ASCII and Unicode

https://en.wikipedia.org/wiki/UTF-8 (ASCII-compatible) Special characters

18-InputOutput/escape.png Newlines

Watch out when moving files and source code from OS to OS!

1.4.3 2. Binary files

For example, ghex /usr/bin/bash:
Binary files typically contain bytes,
that are intended to be interpreted as something other than text characters,
for example image data or executable code.
Compiled computer programs are typical examples;
indeed, compiled applications are sometimes referred to,
particularly by programmers, as binaries.
But binary files can also mean that they contain:
images, sounds, compressed versions of other files, etc.
in short, any type of file content whatsoever.
Even text files can be read in binary mode,
but then you have to interpret the bytes.

1.4.4 Python-specific file handling

What is the syntax and best practice for reading and writing files in python? Files and code for this section

18-InputOutput/file_io_01_dos2unix.py Reading

https://python.swaroopch.com/io.html (general, other stuff too)
https://realpython.com/read-write-files-python/ (good)
https://realpython.com/working-with-files-in-python/ (extensive examples)
https://www.tutorialspoint.com/python3/python_files_io.htm with (context managers)

How does the “context manager” with work, and why should you use it?

1.4.5 Special types of files

These are a couple common file types. Code and file demo

18-InputOutput/employee_birthday.csv csv (a type text file)

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format. A CSV file typically stores tabular data (numbers and text) in plain text, in which case each line will have the same number of fields.
https://automatetheboringstuff.com/2e/chapter16/ pickle (binary and/or text file?)

If you want to save python objects to file, “preserving” or pickling them for later use.
And more from the other links above.

1.4.6 Conclusions

Command line programs and GUI programs can both employ file IO.

Remember the shells we talked about? 02-GitLinuxBash.html

1.5 Standard library modules for file-processing and system interaction

1.5.1 Conclusions

Command line programs and GUI programs can both employ system IO modules for their interaction with the operating system.

1.6 Command line arguments

ls          # is the base program, list
ls -a       # specifies the all option
ls -l       # specifies the long option
ls -a -l    # both options together
ls -al      # same as above, both
ls -al *.py # lists all python files in directiory
man ls      # ls was actually an argument to man

1.6.1 Python command line arguments

The user’s shell splits the command-line string into words before passing them as command-line arguments to Python, which assigns them to sys.argv:

There are many alternatives for writing programs that accept command line arguments upon launch:

1. Manually using sys.argv
2. Python native argparse/getopt.

3. Docopt module
4. Google’s Fire module
5. Click module 0. Overview reading 1. Manual sys.argv

* 18-InputOutput/arg_io_00_args.py
* 18-InputOutput/arg_io_01_manual.py

* This is what you use for 1-off simple stuff, not really production code. 2. Python native argparse/getopt

* https://docs.python.org/3/howto/argparse.html
* https://docs.python.org/3/library/argparse.html
* https://docs.python.org/3/library/getopt.html
* https://www.tutorialspoint.com/python3/python_command_line_arguments.htm
* https://realpython.com/command-line-interfaces-python-argparse/

* 18-InputOutput/arg_io_02_parse.py

* While argparse is standard python, and it can get the job done, it is pretty mediocre, imo. 3. Docopt

* This one is the coolest/best for real command line argument handling, imo!
* It also has ports for many many other languages, so you’d theoretically need to learn one paradigm for writing command line arguments across languages:
* https://github.com/docopt (E.g., C++, Lua, Ruby, Scala, R, and many more)
* The docopt python module is stable and working, but is also stagnant.
* It has some active forks that you can track down on Github if you’re interested.
* Learning docopt is actually a great way to learn the POSIX standard for command line arguments anyway!
* https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap12.html
* and GNU extensions to POSIX conventions:
* https://www.gnu.org/prep/standards/standards.html#Command_002dLine-Interfaces
* https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html
* If anything, docopt becomes a readme standard, in addition to an argument standard, also a good outcome. 4. Fire 5. Click

1.6.2 Conclusions

In my opinion, if you care to efficiently write programs with command line arguments, then learn:
* basic sys.argv usage,
* docopt, and
* fire

+++++++++++ Cahoot-18d-1
How do you access the number of user-defined arguments passed in to a program?
(forgot to show this in screencast…)

Next: 19-TestingFrameworks.html