Redirect data streams with pipes
Using Pipes
Experienced Linux users are probably familiar with the use of pipes. For instance, the following example finds and counts all the subdirectories of the directory where the command is invoked:
$ ls -l | grep "^d" | wc -l 138
The output from a call to ls
is routed to the grep
command's input channel, while the output from grep
itself is routed to the input of the wc
command (Figure 2).
The shell defines which way redirection works for anonymous pipes. It is always from left to right. On completing the command sequence, all processes are terminated and the redirections are revoked.
For some command lines and scripts, you may need pipes for a longer period of time or for multiple processes. For these instances, you can use a named pipe. Named pipes reside in the filesystem, like other files, and persist after a reboot. Multiple processes can use named pipes. There are no restrictions to reading from or writing to the data flow.
Named Pipes
To create a permanent pipe, you can use the mkfifo
command. The example shown in Listing 2 creates a pipe named /var/tmp/testpipe
. The ls
command shows you that the action worked. In the output from ls
(line 3), the p
flag on the left tells you that this is a pipe. You can now redirect the output of a command to this pipe. The command waits before completing processing until another process has read the data from the pipe (Listing 3).
Listing 2
Creating a Named Pipe
01 $ mkfifo /var/tmp/testpipe 02 $ ls -l /var/tmp/testpipe 03 prw-r--r-- 1 root root 0 Jan 4 23:35 /var/tmp/testpipe
Listing 3
Using Named Pipes
### Session 1: Write $ echo "3.1415" >/var/tmp/testpipe ### Session 2: Read $ ls -l /var/tmp/testpipe prw-r--r- 1 root root 0 Jan 4 23:35 /var/tmp/testpipe $ pi=$(cat /var/tmp/testpipe) $ ls -l /var/tmp/testpipe prw-r--r- 1 root root 0 Jan 4 23:42 /var/tmp/testpipe $ echo $pi 3.1415
You can see that the call changes the timestamp from 23:35 to 23:42 after reading. However, the size of the file is still zero bytes because the pipe theoretically only transfers the data. What actually happens, however, is that the operating system provides a buffer, although this is not important for general usage.
A simple example (shown in Listings 4 and 5) clearly demonstrates that – from the processes' perspective – writing to a pipe only starts when the pipe is read. In this session, a command writes the current date and time to the start
variable and then writes the variable's contents to the pipe. It then reads the date and time again and stores the values in the end
variable. The process also immediately writes its value to the pipe. The second session reads both lines from the pipe and outputs them.
Listing 4
Defining Variables
# started=$(date +'%H:%M:%S') ; echo $started # ended=$(date +'%H:%M:%S'); echo $ended 23:04:44 23:04:44
To show that the output is actually generated within next to no time, the first run does not use redirection (Listing 4). Now the two variables are again assigned timestamps and the output is written to the pipe in each case. Shortly after this, the pipe is read line by line in a second session and the results are output onscreen (Listing 5). You can see that there is a difference of several seconds between start and end. The difference results from the fact that the second timestamp is not created until the first line has been read from the pipe.
Listing 5
Reading from a Pipe
### Terminal 1: $ started=`date +'%H:%M:%S'` ; echo $start >/var/tmp/testpipe ; ende=`date +'%H:%M:%S'` ; echo $ended >/var/tmp/testpipe ### Terminal 2: $ read started </var/tmp/testpipe ; read ended </var/tmp/testpipe ; echo "Start: $started" ; echo "End: $ended" Start: 23:08:40 End: 23:08:52
Sample Application
Listing 6 implements a network connection based on named pipes and uses the connection to execute commands on a remote computer.
Listing 6
Remote Command
01 $ ssh [-q] target "command1 [; command2 [;...]]" 02 $ rcmd "command1 [; command2 [;...]]"
The interesting aspect about the communication between two computers is that it does not matter, for the reading or writing process, what happens before or after the pipe, because the read and write actions do not change for the respective process. Listing 6 assumes that access between the two machines is not password protected and relies on SSH in the example.
The ssh
command lets you stipulate both a command chain and the target computer (line 1, Listing 6). SSH now connects to the target system, executes the specified commands, and then terminates the connection. It would be simpler to call a function that is then handed the command to be executed, runs it on the other machine, and prints the output on the local screen (line 2).
The rmcd
function is from the functions
script (Listing 7). Each computer uses one pipe for sending and one pipe for receiving. A process permanently reads the sending pipe and redirects the lines it reads to a data flow. In this example, SSH sends the data flow to the remote computer. On the receiving side, the incoming data flow is redirected to the receiving pipe where it is read. The pipe processes the data that was read and writes the results to the sending pipe in order to send the results back to the originating computer (Figure 3).
Listing 7
The functions Script
001 setenv() { 002 # Set required environment variables 003 awkfile=/var/tmp/read.awk 004 BASENAME=${BASENAME:=/usr/bin/basename} 005 PYTHON=${PYTHON:=/usr/bin/python} 006 AWK=${AWK:=/usr/bin/awk} 007 TAR=${TAR:=/usr/bin/tar} 008 MKFIFO=${MKFIFO:=/usr/bin/mkfifo} 009 UNAME=${UNAME:=/usr/bin/uname} 010 TAIL=${TAIL:=/usr/bin/tail} 011 CAT=${CAT:=/usr/bin/cat} 012 SSH=${SSH:=/usr/bin/ssh} 013 NOHUP=${NOHUP:=/usr/bin/nohup} 014 RM=/usr/bin/rm 015 rhost=${REMOTEHOST:=localhost} 016 lhost=${LOCALHOST:=`$UNAME -n`} 017 sendpipe=/tmp/send.${rhost} 018 receivepipe=/tmp/receive.${rhost} 019 rsendpipe=/tmp/send.${lhost} 020 rreceivepipe=/tmp/receive.${lhost} 021 } 022 023 chkpipes() { 024 # Check whether reuqired pipes exist 025 for pipe in $sendpipe $receivepipe 026 do 027 if [ ! -p $pipe ] 028 then 029 echo "cannot communicate with $rhost" >&2 030 return 1 031 fi 032 done 033 return 0 034 } 035 036 createpipes() { 037 # Generate required pipes 038 setenv 039 for pipe in $sendpipe $receivepipe 040 do 041 if [ ! -p ${pipe} ] 042 then 043 $MKFIFO ${pipe} 044 if [ $? -ne 0 ] 045 then 046 echo "Cannot create ${pipe}" >&2 047 return 1 048 fi 049 else 050 echo "Pipe ${pipe} already exists" 051 fi 052 done 053 return 0 054 } 055 056 removepipes() { 057 # Delete pipes if so desired 058 setenv 059 for pipe in $sendpipe $receivepipe 060 do 061 if [ -p ${pipe} ] 062 then 063 rm -f ${pipe} 064 if [ $? -ne 0 ] 065 then 066 echo "Cannot remove ${pipe}" >&2 067 return 1 068 fi 069 else 070 echo "Pipe ${pipe} does not exist" 071 fi 072 done 073 return 0 074 } 075 076 listen() { 077 setenv 078 chkpipes # Do required pipes exist 079 if [ $? -ne 0 ] 080 then 081 return 1 082 fi 083 # If present delete file with last directory used 084 if [ -w /tmp/lastpwd.${rhost} ] 085 then 086 $RM /tmp/lastpwd.${rhost} 087 fi 088 ( while read line 089 do 090 set $line 091 if [ "$1" = "BEGIN_CMD" ] 092 then 093 shift 094 # Run command 095 # Change to last directory 096 if [ -r /tmp/lastpwd.${rhost} ] 097 then 098 cdircmd="cd `$CAT /tmp/lastpwd.${rhost}` ; " 099 else 100 cdircmd="" 101 fi 102 # Redirect output to data stream to be sent 103 echo "BEGIN_CMD_OUT" >$sendpipe 104 echo "${cdircmd} $@" |/bin/bash >$sendpipe 105 echo "END_CMD_OUT" >$sendpipe 106 elif [ "$1" = "END_COMMUNICATION" ] 107 then 108 # Terminate process if END_COMMUNICATION string is received 109 exit 0 110 elif [ "$1" = "BEGIN_FILETRANSFER" ] 111 then 112 # If a file is to be received 113 filename=`$BASENAME $2` 114 tdir=$3 115 echo "Receiving file $filename..." 116 echo "Copying to $tdir..." 117 # Use awk to read lines until the END_FILETRANSFER 118 # string is received 119 $AWK ' 120 { 121 if ( $0 == "END_FILETRANSFER" ) 122 exit 0 123 else 124 print $0 125 # The received lines are decoded and redirected to the desired target file 126 }' $receivepipe | $PYTHON -c 'import sys,uu; uu.decode(sys.stdin, sys.stdout)' >${tdir}/$filename 127 echo "File $filename transferred" 128 elif [ "$1" = "BEGIN_CMD_OUT" ] 129 then 130 # If BEGIN_CMD_OUT string is received, output all further lines 131 # until END_CMD_OUT is received 132 $AWK ' 133 { 134 if ( $0 == "END_CMD_OUT" ) 135 exit 0 136 else 137 print $0 138 }' $receivepipe 139 fi 140 done <$receivepipe 141 ) & 142 } 143 144 establish() { 145 setenv 146 chkpipes # Do the required pipes exist 147 # Does the awk file exist 148 if [ ! -r $awkfile ] 149 then 150 echo "Cannot find $awkfile" >&2 151 return 1 152 fi 153 if [ $? -eq 0 ] 154 then 155 ( $TAIL -f $sendpipe | ( $SSH -q ${rhost} "$AWK -v pipe=${rreceivepipe} -f ${awkfile}" ) ) & 156 else 157 # If pipes do not exist, quit output and function 158 echo "Pipes for communication not present" >&2 159 return 1 160 fi 161 } 162 163 killall() { 164 # Send end communication to all processes 165 setenv 166 if [ -w $sendpipe ] 167 then 168 echo "END_COMMUNICATION" >$sendpipe 169 echo "." >$sendpipe 170 $RM -f /tmp/sendpipe_${rhost}.pid 171 fi 172 if [ -w $receivepipe ] 173 then 174 echo "END_COMMUNICATION" >$receivepipe 175 $RM -f /tmp/listener_${rhost}.pid 176 fi 177 } 178 179 rcmd() { 180 setenv 181 # Check whether connection to REMOTEHOST exists 182 chkpipes 183 if [ $? -ne 0 ] 184 then 185 return 1 186 fi 187 # Send parsed line 188 echo "BEGIN_CMD $@ ; { pwd >/tmp/lastpwd.$lhost ;}" >$sendpipe 189 return $? 190 } 191 192 sendfile() { 193 setenv 194 # Check whether Python can be executed 195 if [ -x $PYTHON -a ! -d $PYTHON ] 196 then 197 # If Python is not available, report and quit function 198 if [ $# -lt 1 -o $# -gt 2 ] 199 then 200 echo "usage: sendfile FILE [target directory]" 201 return 1 202 fi 203 # By default, copy file to /var/tmp 204 tdir=${REMOTETEMPDIR:=/var/tmp} 205 if [ $# -eq 1 ] 206 then 207 file=$1 208 else 209 file=$1 210 tdir=$2 211 fi 212 # Register and copy file 213 echo "BEGIN_FILETRANSFER $file $tdir" >$sendpipe 214 cat $file |$PYTHON -c 'import sys,uu; uu.encode(sys.stdin, sys.stdout)' >$sendpipe 215 echo "END_FILETRANSFER" >$sendpipe 216 else 217 echo "No python executable found. File transfer not possible." >&2 218 return 1 219 fi 220 }
The communication in Figure 3 is broken down into several segments. Essentially, it is all about receiving data via a data stream, processing the data, and sending back a data stream in response. But how do you maintain a continuous data stream between two computers in order to send commands and other information?
« Previous 1 2 3 4 Next »
Buy this article as PDF
(incl. VAT)