Using PIPEs for Input/Output

HSI supports the use of Un*x "pipes" when either storing or retrieving files, using the standard rename syntax:

    get local_File : HPSS_File


   put local_File : HPSS_File

Using STDIN and STDOUT for Input Output

For "put" commands, if localFIle is a minus sign "-", then HSI reads input from its STDIN.

This is intended for use in shell script pipelines, such as:

         tar cf - . | hsi options "put - : someTarFile"

(Note: HTAR provides the same capability as the above, but is generally much faster).

However, if used interactively, HSI can also be used to read from the TTY to pipe a file into HPSS.  In this case, a single <ctrl-D> (CTRL key and "d" key pressed simultaneously) can be used to terminate the input (take care not to use more than one <ctrl-D>, as this is also a way to exit HSI).  For example:

O:[hpss-nccs]/home/gleicher/temp->put - : ToDo-Today

Here is my TO-DO list:   (Lines entered from the TTY)

       wake up                



       go to bed


put  '<stdin>' : '/home/gleicher/ToDo-Today' ( 78 bytes, 0.0 KBS (cos=5081))

For "get" commands, if localFile is a minus sign, then HSI pipes the output to its STDOUT. Again, this is intended for use in shell script pipelines, such as:

         hsi options get - : someTarFile | tar xf -

As explained in HSI Special Topics, Tips and Tricks, this is one way to view the contents of HPSS files without first copying them to a local file, for example, using the above HPSS file:

       get - : ToDo-Today

Here is my TO-DO list:            

       wake up



       go to bed

get  '<stdout>' : '/home/gleicher/ToDo-Today' (2010/12/20 19:47:38 78 bytes, 221.7 KBS )

Using Un*x PIPE Commands for Input and Output

For both storing and retrieving, if the first character of localFile is a vertical bar "|", HSI interprets the remainder of localFile as a pipe command after skipping any leading white space.  

The pipe command can contain one or more shell commands, such as:

        put "| cat file1.c file2.c " : oneCFile


        put "| ls -l | awk '{print $9}' " : listOfFilenames


       get "|tar xf - " : tools.tar   


       get "|less" : bigDocument  (see note 3 and 4)


1.  "localFile"  must be enclosed in quotes if it contains whitespace characters.

In addition, if this form is used on a shell command line,  and “localFile” contains whitespace characters, then the quoted “localFile” string must itself be enclosed in a separate set of quotes in order to ensure that HSI sees the quotes around “localFile”

when it parses the get or put options.

Normally a single quote is used to quote the double-quoted “localFile” string.

For example, this will work:

     hsi options put ‘“| ls -a “‘ : listingFile  

but this will fail:

    hsi options put “| ls -a “ : listingFile

The reason is that in the first form, the shell parses the command line and passes

the string “| ls -a” as a paramter to HSI, including the double quotes, so HSI sees the

command parameters as:

   1. the string (including quotes)  “| ls -a”

   2. the colon string (:)

   3. the string listingFile

In the second form,the shell removes the double quotes, and HSI sees the options as 5


  1. |

  2. ls

  3. -a

  4. :

  5. listingFile

In the second form, HSI interprets 1 and 2 as local filenames, and 3,4 and 5 as

the local : HPSS form, withlocal file -a and HPSS file.

2.  To disable HSI file transfer progress messages, which can overwrite parts of the last line of output on the TTY, use either the "PROGRESS" command toggle, or the "-q" (quiet) option on the HSI command line.
3. When using interactive utilities such as "less", it is usually, although not always, necessary to use "exit" (or "end", "quit", or whatever the utility recognizes as a command to terminate) in order to exit the utility, even if HSI is able to pipe the entire file to the utility.  

WARNING:  HSI transfers entire files, using the parallel transfer features of HPSS in 

order to achieve the fastest possible transfer rates.  When reading HPSS files to STDOUT, or to a local pipe command, it is possible to cause a transfer to abort if the

file is longer than two data buffers (8 MB by default), and data is not consumed at a fast enough rate.  This is normally not a problem for utilities such as “ " or "md5", but

could potentially be a problem for tty-oriented utilities such as "less", "more", "tail",


For information on how HSI chooses an HPSS Class of Service when reading from a Un*x pipe, see the page “HSI COS Selection With Piped Files