Co:Z SFTP supports recording SMF type 119 records that are compatible with the following IBM FTP records:
Subtype 100 - FTP server transfer initialization (real-time SMF data NMI only)
Subtype 101 - FTP client transfer initialization (real-time SMF data NMI only)
Refer to the z/OS Communications Server: IP Programmer's Guide and Reference for complete documentation on FTP SMF type 119 records. Section SMF Record Formats below highlights Co:Z SFTP specific field information.
In addition to standard FTP completion/initialization records above, Co:Z SFTP also creates the following SMF 119 record subtypes:
Subtype 194 - Co:Z SFTP server interim transfer (real-time SMF data NMI only)
Subtype 195 - Co:Z SFTP client interim transfer (real-time SMF data NMI only)
For more information on the Co:Z SFTP specific type 119 records, see section SMF Record Formats.
Note: Record types 100, 101, 194, and 195 are never written as real SMF records
In order to enable recording of Co:Z SFTP SMF 119 records, you must:
configure SMF to allow recording these records and subtypes. See z/OS MVS System Management Facilities (SMF) for more information.
permit the users running Co:Z SFTP client or server jobs
READaccess to theBPX.SMFFACILITYclass resource. Alternatively, you may also use type/subtype specific permissions (see next section).the
nosmfconfiguration option must not be set. See http://coztoolkit.com/docs/sftp/options.html#options_misc for more information.in order to get accurate local and remote host/port information for client SMF records, the program
COZ_HOME/bin/ssh-socket-infois called by Co:Z once the child ssh session is established.This program uses the IBM EZBNMIFR network management API, which requires the ssh-socket-info program to be APF authorized. The Co:Z installer will attempt to set the "+a" extattr bit on this program, but will only succeed if the installing userid has READ access to the BPX.FILEATTR.APF SAF resource. If for some reason, this program is not APF authorized, Co:Z SFTP will operate properly, but the SMF socket information will not be accurate in client SMF records.
Introduced by APAR OA48775, z/OS now allows non-authorized programs to write specific SMF record types/subtypes. This is supported starting with Co:Z SFTP 4.5.0 using the following steps:
Permit the users running Co:Z SFTP client or server jobs
READaccess toBPX.SMF.119.nresource, for n = {3, 70, 192, 193}.The Co:Z SFTP client and server programs must be program controlled. Starting in release 4.5.0, The Co:Z installer will attempt to set the "+p" extattr bit on the Co:Z SFTP client and server programs (
cozsftp_cmdandsftp-server) in the install directory.For running the Co:Z SFTP client in batch, you must explicitly mark the
COZ.LOADLIBdataset as program controlled. If you are using Co:Z SFTP server user exits, this load library must also be marked as program controlled.The address spaces where you run Co:Z SFTP must remain program-controlled "clean" - in other words, you may not run any non-program controlled commands in the same address space prior to running Co:Z SFTP:
For Co:Z SFTP server, do not run any non-program controlled commands in your system or user-level
sftp-server.rcscripts. Commands may be run using$(cmd ...)or`cmd ...`or by temorarily usingexport _BPX_SHAREAS=NOand back toYESaround the command, since these will not run in the same address space.For Co:Z SFTP client, watch for commands that might run in the script that you use to invoke the
cozsftpcommand, or in the/etc/profileor$HOME/.profilescripts. Starting in 4.5.0, the sampleSFTPPROCwill start the z/OS shell in the same address space but with_BPX_SHAREAS=NO. Any commands issued by the profile scripts prior setting_BPX_SHAREAS=YESwill run in a separate address space to avoid dirtying the program-controlled environment.To diagnose program control issues in client batch jobs, run the step with:
ARGS='-LD /bin/sh -Lx'to enable COZBATCH and z/OS shell tracing.
The Co:Z SFTP client and server will also write SMF 119 records to a Unix datagram socket
if it is available.
By default, the name of the socket is /var/log/cozsftp.smf.sock unless overridden
by the SFTP_SMF_SOCK environment variable.
This interface is useful in managed file transfer environments that need real-time
access to file transfer events. The real-time interface is independent of actual SMF recording -
you may use either real SMF recording, the datagram socket, or both.
SMF 119 record subtypes related to interim file transfer logging are only written using the real-time Co:Z SMF Interface.
Real-time logging of these records is enabled by setting the option intermlogging=nnnn where nnnn is the interval
in seconds. Interim log messages are written during a file transfer. When this feature is enabled and a file transfer is
initiated, an initialization record is written at the start of the transfer
(subtype 100 by the Co:Z SFTP server,
subtype 101
by the Co:Z SFTP client). At the interval specified, interim records
(subtype 194 by the Co:Z SFTP server,
subtype 195
by the Co:Z SFTP client) are logged capturing the bytes transferred at the time identified in the record header.
See Miscellaneous options.
To use this facility, you must write a program that creates this Unix-domain socket and
receives datagram messages from it.
Each message will be a SMF record image from a Co:Z SFTP client or server running on the same system.
A sample C++ program, CoZSmfServer.C, demonstrates
how to use this facility. See the documentation and build instructions in
$COZ_HOME/samples/smfapi/CoZSmfServer.C. This sample illustrates the following scenarios:
consolidation of BPX.SMF authorization to a single job or user, passing of SMF records in real-time to another program,
and real-time logging of initialization, interim and completion file transfer SMF records.
The z/OS Communications Server: IP Programmer's Guide and Reference contains complete documentation on FTP SMF type 119 records. This section highlights Co:Z SFTP specific field information (shown in bold) as well as record formats for Co:Z SFTP type 119 subtypes.
TCP/IP identification
Offset Length Format Description 0 8 EBCIDIC System name 8 8 EBCIDIC Sysplex name 16 8 EBCIDIC TCP/IP stack name 24 8 EBCIDIC TCP/IP release identifier. Set to '011100' for V1 Release 11. 32 8 EBCDIC TCP/IP subcomponent. Set to 'SFTPS' (SFTP server) or 'SFTPC' (SFTP client). 40 8 EBCDIC ASName 48 8 EBCDIC UserID 56 4 binary ASID 60 1 binary Reason. Set to X'08', Event SMF record. 61 3 binary reserved FTP security
Offset Length Format Description 0 1 EBCIDIC Protection Mechanism. Set to T: TLS. 1 1 EBCIDIC Control Connection Protection Level. Set to P: Private. 2 1 EBCDIC Data Connection Protection Level. Set to P: Private. 3 1 EBCDIC Login Method. Set to P: Password. 4 8 EBCDIC Protocol level. Set to blanks. 12 20 EBCDIC Cipher Specification. Set to blanks. 32 4 EBCDIC Protection buffer size. Set to 0. 36 2 binary Reserved
Self defining section
The self-defining section identifies 6 triplets, although 7 are allocated. The triplets are:
TCP/IP identification
FTP client transfer completion
FTP client transfer completion associated data set name
FTP client SOCKS - triplet set to zero
FTP security
FTP user name
FTP client transfer completion
Several fields noted below are set from ssh socket information, if available. See section Enabling SMF recording for additional information.
Offset Length Format Description 0 4 EBCIDIC FTP command 4 4 EBCIDIC Local file type 8 16 binary Remote IP address (data connection). Set from ssh socket information, if available. 24 16 binary Local IP address (data connection). Set from ssh socket information, if available. 40 2 binary Local port (data connection). Set from ssh socket information, if available. 42 2 binary Remote port (data connection). Set from ssh socket information, if available. 44 16 binary Remote IP address (control connection). Set equal to the data connection value. 60 16 binary Local IP address (control connection). Set equal to the data connection value. 76 2 binary Remote port (control connection). Set equal to the data connection value. 78 2 binary Local port (control connection). Set equal to the data connection value. 80 8 EBCIDIC Server user id 88 8 EBCIDIC Local user id 96 1 EBCIDIC Data format 97 1 EBCIDIC Transfer mode 98 1 EBCIDIC Structure 99 1 EBCIDIC Data set type 100 4 binary Transfer start time 104 4 packed Transfer start date 108 4 binary Transfer end time 112 4 packed Transfer end date 116 4 binary Transfer duration 120 8 binary Transmission byte count 128 4 EBCIDIC Last server reply 132 8 EBCIDIC PDS member name 140 8 EBCIDIC Host name 148 8 EBCIDIC Abnormal end information 156 8 floating point hex Transmission byte count (float) 164 4 binary TCP connection ID (control connection). Set from ssh socket information, if available. 168 4 binary TCP connection ID (data connection). Set equal to the control connection value.
Self defining section
The self-defining section identifies 6 triplets, although 7 are allocated. The triplets are:
TCP/IP identification
FTP server transfer completion
FTP server host name
FTP server first associated data set name
FTP server second associated data set name
FTP security
FTP server transfer completion
Offset Length Format Description 0 1 binary FTP operation 1 3 binary reserved 4 4 EBCIDIC FTP command 8 4 EBCIDIC Local file type 12 16 binary Remote IP address (data connection) 28 16 binary Local IP address (data connection) 44 2 binary Local port (data connection) 46 2 binary Remote port (data connection) 48 16 binary Remote IP address (control connection). Set equal to the data connection value. 64 16 binary Local IP address (control connection). Set equal to the data connection value. 80 2 binary Remote port (control connection). Set equal to the data connection value. 82 2 binary Local port (control connection). Set equal to the data connection value. 84 8 EBCIDIC Client user id on server 92 1 EBCIDIC Data type 93 1 EBCIDIC Transmission mode 94 1 EBCIDIC Data Structure 95 1 EBCIDIC Data set type 96 4 binary Transfer start time 100 4 packed Transfer start date 104 4 binary Transfer end time 108 4 packed Transfer end date 112 4 binary Transfer duration 116 8 binary Transmission byte count 124 4 EBCIDIC Last reply to client 128 8 EBCIDIC PDS member name 136 8 EBCIDIC Abnormal end information 144 8 EBCIDIC Second PDS member name 152 8 floating point hex Transmission byte count (float) 160 4 binary TCP connection ID (control connection). Set to 0. 164 4 binary TCP connection ID (data connection). Set to 0. 168 15 EBCIDIC Session id. Set to a generated value: jobname followed by the last five digits of the process id. 183 1 binary reserved
Real-time transfer SMF records are not written by default.
Refer to Miscellaneous options for information on setting the interimlogging
option to enable this feature. Additionly, see Using the
real-time Co:Z SMF interface for information on accessing real-time SMF records.
Self defining section
The self-defining section identifies 6 triplets, although 7 are allocated. The triplets are:
TCP/IP identification
FTP server transfer initialization
FTP server host name
FTP server first associated data set name
FTP server second associated data set name
FTP security
FTP server transfer initialization
Offset Length Format Description 0 1 binary FTP operation 1 1 binary Passive or active mode data connection. Set to X'00': Active using default IP and port. 2 2 binary reserved 4 4 EBCIDIC FTP command 8 4 EBCIDIC Local file type 12 16 binary Remote IP address (data connection) 28 16 binary Local IP address (data connection) 44 2 binary Local port (data connection) 46 2 binary Remote port (data connection) 48 16 binary Remote IP address (control connection). Set equal to the data connection value. 64 16 binary Local IP address (control connection). Set equal to the data connection value. 80 2 binary Remote port (control connection). Set equal to the data connection value. 82 2 binary Local port (control connection). Set equal to the data connection value. 84 8 EBCIDIC Client user id on server 92 1 EBCIDIC Data type 93 1 EBCIDIC Transmission mode 94 1 EBCIDIC Data Structure 95 1 EBCIDIC Data set type 96 4 binary Data connection start time. Set to the start time of the session. 100 4 packed Data connection start date. Set to the start date of the session. 104 4 binary Control connection start time. Set equal to the data connection value. 108 4 packed Control connection start date. Set equal to the data connection value. 112 8 EBCIDIC PDS member name 120 8 EBCIDIC Second PDS member name 128 4 binary TCP connection ID (control connection). Set to 0. 132 4 binary TCP connection ID (data connection). Set to 0. 136 15 EBCIDIC Session id. Set to a generated value: jobname followed by the last five digits of the process id. 151 1 binary reserved
Real-time transfer SMF records are not written by default.
Refer to Miscellaneous options for information on setting the interimlogging
option to enable this feature. Additionly, see Using the
real-time Co:Z SMF interface for information on accessing real-time SMF records.
Self defining section
The self-defining section identifies 6 triplets, although 7 are allocated. The triplets are:
TCP/IP identification
FTP client transfer initialization
FTP client associated data set name
FTP client SOCKS - triplet set to zero
FTP security
FTP user name
FTP client transfer initialization
Several fields noted below are set from ssh socket information, if available. See section Enabling SMF recording for additional information.
Offset Length Format Description 0 4 EBCIDIC FTP command 4 4 EBCIDIC Local file type 8 16 binary Remote IP address (data connection). Set from ssh socket information, if available. 24 16 binary Local IP address (data connection)Set from ssh socket information, if available. 40 2 binary Local port (data connection)Set from ssh socket information, if available. 42 2 binary Remote port (data connection)Set from ssh socket information, if available. 44 16 binary Remote IP address (control connection). Set equal to the data connection value. 60 16 binary Local IP address (control connection). Set equal to the data connection value. 76 2 binary Remote port (control connection). Set equal to the data connection value. 78 2 binary Local port (control connection). Set equal to the data connection value. 80 8 EBCIDIC Server user id 88 8 EBCIDIC Local user id 96 1 EBCIDIC Data format 97 1 EBCIDIC Transfer mode 98 1 EBCIDIC Structure 99 1 EBCIDIC Data set type 100 4 binary Start time of data connection. Set to the start time of the session. 104 4 packed Start date of data connection. Set to the start date of the session. 108 4 binary Start time of control connection. Set equal to the data connection value. 112 4 packed Start date of control connection. Set equal to the data connection value. 116 8 EBCIDIC PDS member name 124 1 EBCIDIC Passive or active mode data connection. Set to X'00': Active using default IP and port. 125 3 binary reserved 128 4 binary TCP connection ID (control connection). Set from ssh socket information, if available. 132 4 binary TCP connection ID (data connection). Set equal to the control connection value.
Self defining section
The self-defining section identifies 3 triplets, although 7 are allocated. The triplets are:
TCP/IP identification
Socket connection
Co:Z SFTP messages
Socket connection
Offset Length Format Description 0 16 binary Remote IP address 16 16 binary Local IP address 32 2 binary Remote port number 34 2 binary Local port number 36 15 EBCDIC FTP session ID. Set to a generated value: jobname followed by at most the last five digits of the process id. 51 1 binary reserved Co:Z SFTP messages
This section contains Co:Z SFTP messages, informational level or above, that were associated with the previous transfer. One or more message sub-sections may be included, each with the following layout:
Offset Length Format Description 0 4 binary Time (in local time) 4 4 Packed Date (in local time) 8 2 binary Length of message that follows 10 variable EBCDIC Message text
Self defining section
The self-defining section identifies 3 triplets, although 7 are allocated. The triplets are:
TCP/IP identification
Socket connection
Co:Z SFTP messages
Socket connection
Fields noted below are set from ssh socket information, if available. See section Enabling SMF recording for additional information.
Offset Length Format Description 0 16 binary Remote IP address. Set from ssh socket information, if available. 16 16 binary Local IP address. Set from ssh socket information, if available. 32 2 binary Remote port number. Set from ssh socket information, if available. 34 2 binary Local port number. Set from ssh socket information, if available. 36 15 EBCDIC FTP session ID. Set to blank. 51 1 binary reserved Co:Z SFTP messages
This section contains Co:Z SFTP messages, informational level or above, that were associated with the previous transfer. One or more message sub-sections may be included, each with the following layout:
Offset Length Format Description 0 4 binary Time (in local time) 4 4 Packed Date (in local time) 8 2 binary Length of message that follows 10 variable EBCDIC Message text
Real-time transfer SMF records are not written by default.
Refer to Miscellaneous options for information on setting the interimlogging
option to enable this feature. Additionly, see Using the
real-time Co:Z SMF interface for information on accessing real-time SMF records.
Self defining section
The self-defining section identifies 7 triplets. The triplets are:
TCP/IP identification
FTP server transfer initialization - Set equal to FTP server transfer initialization (subtype 100)
FTP server host name
FTP server first associated data set name
FTP server second associated data set name
FTP security
FTP interim transfer
FTP interim transfer section
Offset Length Format Description 0 8 binary Estimated file size (bytes). Set to -1 on put (write) or if read and source file size is unknown. 8 8 binary Estimated file size (bytes float). Set to -1 on put (write) or if read and source file size is unknown. 16 8 binary Interim transmission byte count 24 8 floating point hex Interim transmission byte count (float)
Real-time transfer SMF records are not written by default.
Refer to Miscellaneous options for information on setting the interimlogging
option to enable this feature. Additionly, see Using the
real-time Co:Z SMF interface for information on accessing real-time SMF records.
Self defining section
The self-defining section identifies 7 triplets. The triplets are:
TCP/IP identification
FTP client transfer initialization - Set equal to FTP client transfer initialization (subtype 101)
FTP client associated data set name
FTP client SOCKS - triplet set to zero
FTP security
FTP user name
FTP interim transfer
FTP interim transfer section
Offset Length Format Description 0 8 binary Estimated file size (bytes). Set to -1 on get (write) or if read and source file size is unknown. 8 8 binary Estimated file size (bytes float). Set to -1 on get (write) or if read and source file size is unknown. 16 8 binary Interim transmission byte count 24 8 floating point hex Interim transmission byte count (float)
