FTP and SFTP vs. MFT for OS/400, IBM i, platforms

8 03 2010

Over the past few weeks, I have seen a lot of news group chatter regarding FTP, FTPS and SFTP relating to the IBM System i, i/OS. Although FTP(S) and SFTP provide workable options when limited file transfers are need, they lack the functionality and usability of a mature Managed File Transfer (MFT) solution. Let’s look at some of the advantages provided by a good MFT product verses FTP.

For the purpose of this post, the term OS/400 also refers to i5/OS, i/OS and IBM i.

Under OS/400, SFTP is provided via the PASE and its use is described in this IBM Systems magazine article.

FTP(S)/SFTP vs. MFT functionality

Transferring nested directories is time consuming without a good GUI interface. MFT solutions provide simple and easy-to-use methods for transferring nested directories.
FTP(S)/SFTP only provides two party transfers. MFT allows three party transfers. In a two party transfer, files are transferred between the server and the client. In a three party transfer, the client sets up transfers between two servers so that an intermediate transfer is not necessary.
With FTP(S)/SFTP, controlling end-of-line can be tricky at best. MFT provides straight forward means by which to specify the character or character sequence wanted for end-of-line.
Using OS/400 FTP(S)/SFTP, you may have to create files before doing the transfer to get the correct file settings. An advanced MFT product allows the user to set file appropriate attributes before the transfer or detect those attributes in an OS/400 to OS/400 file transfer. Also, a good product provides one or more methods for automated file creation for save files and database files requiring DDS.
FTP only provides basic scripting. Advanced MFT products provide a full fledged scripting language allowing automation of even the most sophisticated transfer processes.
FTP on OS/400 allows execution of simple commands. Modern, full function MFT products provide the ability, possibly via add-on technology, to not only execute OS/400 commands, but also commands on other systems. A really advanced product also provides logging and control options for the remote system.
OS/400 FTP allows setting a CCSID when opening the FTP session. MFT products go beyond initial CCSID settings by detecting and automatically setting the CCSID for each file transferred during a multiple file transfer whether transferring from the QSYS or IFS file system. A really great MFT product will also adjust end-of-line settings based on ASCII vs. EBCDIC file type.
SFTP only provides binary transfers. FTP supports Single Byte Character Set, SBCS, code pages and some FTP products support UTF-8 code pages. Cutting edge MFT products may support all of the Unicode variants as well as Double Byte Character Set code pages. Although the author knows of none, MFT products that fully supports Mixed Byte Character Sets may exist.
FTP(S) and SFTP provide limited, if any, fault tolerance. MFT products provide network fault tolerance allowing transfer completion following network connection failure and recovery. They may also provide manager fault tolerance for remote command execution whereby remote commands may complete during network outages. Following network recovery, output from reconnected processes is transferred back to the initiating system.

The above information primarily addresses MFT functionality; however, all of the functionality potentially included in an MFT product is not covered. Look for such things as the ability to move files as opposed to only copying files and the ability to list files to name only a couple of items. Security options are referred to but not discussed in detail since they are limited in regard to FTP. The issue of data integrity was not discussed and should be carefully considered before purchasing an MFT product.


Is the IBM i secure or irrelevant? Vulnerabilities

4 02 2010

In the past ten years, IBM dedicated a lot of resources to updating IBM i. With large increases in functionality, including Web enablement, comes additional opportunities for profit as well as venerability. Many traditional business users are not interested in improvements in areas such as Web enablement and, therefore, will not be exposed to many of the vulnerabilities. However, to ignore potential problem areas can result in exposure to regulatory violations and legal actions.

This post from Jon R. Kibler, Chief Technical Officer at Advanced Systems Engineering Technology, Inc. provides some insight into the IBM i vulnerabilities. Although the list is quite small and I expect that a number of them have been addressed, the point is that all system platforms, including IBM i, have vulnerabilities. If there is no one in-house that can establish audit requirements, set up user guidelines, provide substantive policy and tailor your system, or systems, to implement those requirements, guidelines and policies, then thought should be given to bringing in an outside security consultant.

Other ways of limiting your exposure include the use of third party functionality such as Yahoo! Merchant Solutions which can provide a secure customer front end with the ability to transfer orders to your business system for processing, the use of virtual cloud technology such as the Rackspace Virtual Private Cloud and the use of highly secure, cross platform managed file transfer technologies from companies such as Stonebranch. Of course, these are only three options out of thousands that exist.

Is the IBM i secure or irrelevant? 16 bit pointers

31 12 2009

In my previous post, I pointed out that the internet is the most complex system in existence. I also promised insight regarding security issues relating systems attached to the internet. Although the first and most important way to prevent security breaches is to keep your system patches, PTFs, fixes, etc. up to date, the subject of this post is IBM i (i/OS).

I see a lot of articles and discussions about whether IBM i is really secure or just so old that it’s really irrelevant. In point of fact, it’s very difficult to find any hard information explaining the how or why of IBM i security or lack thereof. To treat the subject fairly requires a book, so I will only address one small aspect of IBM i security today – 16 bit pointers.

First, 16 bit pointers have what’s called tags associated with them. That makes the pointers actually 18 bits with two bits that only the kernel (System Licensed Internal Code) can access. If these pointers are changed outside of a kernel operation, one or both of the tag bits get reset which invalidates the pointer. Although it’s not apparent to high level languages, the system is designed so that all information, on disk or in main storage, is accessed via these pointers. As a result, all information is addressed as though residing in main storage; hence, the term single level storage.

The system also organizes storage and uses the pointers in such a way that a program cannot go outside of the bounds of its assigned storage without causing a system exception. In addition, no program, procedure or data object – files, data spaces, queues, etc. – gets accessed without full kernel validation using these pointers. This validation includes access, operation and management rights verification against the user profile whose authority rights are being used to access the object. To protect the system against outside manipulation, certain system critical objects, including user profiles, cannot exist outside of the system Auxiliary Storage Pool.

Due to the design of these pointers and their use by the kernel, a worm or virus cannot take control of a program and trash or corrupt the kernel or access unauthorized information so long as the system runs at security level 40 or above. More about security levels at another time.

This only touches on one aspect of IBM i security. However, the security of the most security capable system in the world depends on
1. The competency of the system administrators
2. Limiting user authorities to only those needed to perform the required jobs and
3. Properly securing assets including passwords.

To dig deeper into aspects of pointers and IBM i internals, read the following. Some of the articles are outdated but still contain important information. Also, some of the articles contain specific information that enhances understanding of IBM i in addition to repeated information from other articles.

Notes for Storage Research by David McKenzie

AS/400 Memory Management from Inside the AS/400 by Frank G. Soltis

Architecture of the IBM AS/400 by Vincent LeVeque at James Madison University, document CS511 dated March 2001

An Object-based Systems Architecture from Smalltalk Musings by Richard Demers

Use the MI to Work with Pointers in ILE RPG by Junlei Li in MC Press Online

The AS/400’s Architecture Goes Beyond Technology by Frank G. Soltis via SystemiNetwork

OS/400 Storage Management from the IBM Information Center

Materialize Pointer (MATPTR) from the IBM Information Center

Resolve System Pointer (RSLVSP) from the IBM Information Center

Where’s my File?

19 09 2009

When most IT folks ask the question, “Where’s my file?”, they mean text files, executable files, DLL files, INI files, and many other types of files. However, that question is just a “little” more complicated on the IBM i platform. “How so?” To IBM i, everything’s an object. There are program objects, class objects, job description objects, subsystem objects and many more object types. And, file objects are actually made up of multiple system objects. “What does that have to do with file transfer?” IBM i file transfer programs normally only handle database files, save files and physical source files which are actually a type of database file.

“As long as I get my data, why should I care?” IBM i objects, as with modern languages, have attributes associated with them. These attributes establish record length, maximum number of file members, access path characteristics, sorting sequence, authority inheritance, data definition specification source file name and many other things. “Why do so few managed file transfer programs handle IBM i attributes?” To do so the file transfer program must be aware of file object attributes and their associated fields. Handling file object attributes requires that file transfer programs carry attribute information as part of the file transfer process. Providing this level of sophistication requires a level of technical expertise unavailable to most companies and a high commitment to customer satisfaction.

“So what?” To make up for the file transfer program’s inability to properly handle file attributes and fields, users have to spend a lot of ‘wasted’ time setting up files prior to the actual transfer. They must then transfer the files in binary which does not allow text field translations. Without a sophisticated IBM i file transfer program, transferring thousands, or even hundreds, of files a day can quickly become prohibitive or require a lot of special automation.

Is “Where’s my file?” the only question to ask? What about where’s the meta-data, audit trails and process visibility? There’s also how do I cost effectively handle multiple-platform and multiple-system configurations. These are critical questions for companies with thousands, hundreds or even tens of of systems that may be spread around the globe.

Stdio problem when moving non-i5 user to native processes via ssh

3 09 2009

We needed to move non-i5 internal users to native i5 processes. For this to work effectively I set links to processes under the QShell because it provides direct native execution and then set up PASE scripts to run via ssh. A number of IBM documents describe setting up ssh so that users need never see a native i5 screen.

Example application – SSH

OpenSSH: How to Stop SSH from Creating Thousands of Job Logs

IBM Redbook
Securing Communications with OpenSSH on IBM i5/OS, July 2006

‘Qshell and OpenSSH for IBM System i’ by Bob Bittner dated 15-May-2007
Note: This paper is an excellent introduction to the QShell.

The problem is that QP2SHELL does not properly handle stdio for the QShell and QP2TERM cannot be used because the job running under the SSH subsystem is not interactive. Using QSH CMD(‘usr/sbin/sshd’) as described in Bob Bittner’s document in place of ‘CALL PGM(QP2SHELL) PARM(”/QOpenSys/usr/sbin/sshd”)’ – note that ” is two single quotes – , does correctly handle QShell stdio. So, changing the follwing

TEXT(‘Job description for SSHD autostart’) USER(SSHDUSR)
RQSDTA(‘CALL PGM(QP2SHELL) PARM(”/QOpenSys/usr/sbin/sshd”)’)


TEXT(‘Job description for SSHD autostart’) USER(SSHDUSR)
RQSDTA(‘QSH CMD(”/QOpenSys/usr/sbin/sshd”)’)

resolves the problem. The command ‘QSH CMD(”/usr/sbin/sshd”)’ may also be used as both /QOpenSys/usr/sbin/sshd and /usr/sbin/sshd link to the same executable.

One additional problem remains; how to get the stdio from the various environments, native, QSHELL and PASE to all display in the ssh terminal window. This is accomplished via either the .env or .profile file. See http://www.uic.edu/depts/accc/software/unixgeneral/unixcust.html and http://publib.boulder.ibm.co/infocenter/systems/scope/aix/index.jsp for additional information regarding these files. To obtain the needed environmental variables and their settings, invoke PASE using CALL QP2TERM from a native command line and enter env. Use the output from the env command for the .profile file since the installed ssh client uses the BSH shell. Do not include variables that will result in conflicts with the shell such as the SHELL, LOGNAME, LANG or HOME.

I have implemented and tested this on both V5R3 and V5R4 systems.

It should be noted that there are many instances where the system command from PASE should be used to execute native processes rather than using QShell. In fact, I used both methods in the PASE shells implemented for this project.

Some may also find the following IBM Software Technical document, OpenSSH: Configuring Server / Client, helpful.

Moving big files securely on the iSeries is tough stuff

19 08 2009

I pen this post as an OS/400 (i5/OS, IBM i) specialist – my first blog post ever.  I ultimately spend a good deal of my time optimizing our products for IBM i and, as such, I’ve found some interesting things and created some better ways to move files on a 400.  When doing OS/400 to OS/400 file transfers, Infitran automatically detects appropriate save file, physical file, or stream file attributes, including CCSID and source DDS, and creates the target file with the same attributes without the need to create the file ahead of time. When transferring files from other platforms, users may either accept default attributes or specify them for the transfer. All of this can be easily automated via the scripting engine.

Managed File Transfer for the iSeries provides both security and functionality that far exceed FTP in management and security. The benefits of MFT is really about event management and visibility.  I look forward to collaborating with the readers and IBM i users to improve our products and to learn more about IBM i in general.