Essential CLI Tools for MacOS X

Version 0.29, 20020903

This work is copyright (c) 2001, 2002 by Tat-Chee Wan, with input and corrections from other contributors. It can be reproduced in part or in full as long as appropriate credit is given to the author(s).

(Author's note: This is a work in progress. It's short on specifics in some areas at this moment, as I haven't had time to dig out the references and fill in the details)

This document is written by someone who's familiar with Unix (RedHat Linux, primarily), who was familiar with MacOS 7.5 (a former Mac IIsi owner), who tolerates Windows 9x only as much as needed to get stuff done, and is now struggling with getting MacOS X to behave. I suppose if I were familiar with FreeBSD or other BSD-variants, it might be a little easier adjusting to MacOS X, but I've always perceived that documentation for configuring BSD-variants as lacking compared with documentation for configuring Linux (with the Linux Documentation Project and its excellent HOWTOs collection). Of course, MacOS X has its own quirks that is not addressed satisfactorily by any of the current Unix documentation sources.

The following are excerpts from my journey of discovery into MacOS X. Hopefully it'll shorten the learning curve for others faced with the same issues.

P.S. The information here should be valid for OS X 10.1.x. I haven't used OS X 10.2 and don't foresee doing so in the near future (due to budget issues).

Changelog

Date Version Comment
20020903 0.29 Added link to Apple KBase article on Deleting Locked Files
Noted macbin only available on OSXS
20020301 0.28 Directory Copying using rsync
Added link to Mark Swift's OSX Server Technical Notes
and Wilfredo Sánchez's talk at USENIX 2000
Notes on Server Updates and Versioning
20011204 0.27 Multiple Server Names after Reconfiguration
MvMac info, Apple Bug Reporting URL,
macbin caveat
20011128 0.26 Corrections to Macintosh Files section
20011121 0.25 Initial Public Release

Contents

  1. Introduction
  2. Partitioning
  3. Macintosh Files
  4. User Accounts
  5. Server Services
  6. Updating Software and Server Versions
  7. Resources
  8. Acknowledgements

Introduction

Macintosh OS X is at its core, a BSD-variant Unix system called Darwin. Aqua, the MacOS X GUI provides transparent access to various Macintosh specific features, such as Resource forks, HFS file attribtes, etc. Normally, users won't have to contend with Darwin, and would be able to work happily via the MacOS X Finder and Systems Preference to configure MacOS X. However, for Unix-geeks like myself, this is not sufficient. How can one ignore the large body of Open Source software that is just waiting to be downloaded and installed?

First, some caveats. Don't believe everything you read in the man pages in OS X. The man pages were ported wholesale from FreeBSD, and has not been updated to reflect the altered state of reality that is MacOS X. Second, configuration files are not always stored under /etc, despite the Darwin/Unix roots.

In addition, while Mac OS X and Mac OS X Server are essentially the same core OS, the behavior of both systems can differ. Notably, in terms of provided administrative tools (e.g., Server Admin is only available for OS X Server). When appropriate, distinction would be made.

Glossary

OS9: Mac OS 9.x (Classic)
OSX: Mac OS X 10.x (all versions)
OSX.1: Mac OS X 10.1
OSXS: Mac OS X Server 10.x (all versions)
OSXS.1: Mac OS X Server 10.1

Partitioning

This is perhaps the most poorly documented setup step in OS X. There's also conflicting information provided by Apple's Developer Connection website on this topic (number of UDF partitions). I couldn't find any good reference to the topic that describes all that I wanted to know. Most discussion centers on whether to partition for OS 9.x and OS X, etc. However, since OS X is a Unix-based OS, partitioning is crucial for maintaining system sanity, especially in disk full situations or in case of file system crash. The rule of thumb is that you should leave at least 10% of the disk space free for vital Unix partitions (e.g., root '/', /var, /tmp) in order to avoid unusual system behavior. This is especially true if multiple users are allowed unlimited disk storage in their accounts.

The typical partitioning scheme in Linux/Unix is to have separate partitions for the following:

However, for OS X, the default partitioning scheme is to have a single huge HFS+ partition for the entire drive. This is not a good thing for the reasons mentioned previously. To make things more interesting, OS X disk partitions do not have fixed unique physical device names (/dev/disk0sYY). Instead, partitions are mounted on boot, therefore a given partition can be mounted to different devices depending on the order of drive spinup etc. [This was discussed in the DarwinOS Users lists, among other places]. Even more confusing, the mount command for mounting disks does not appear to refer to /etc/fstab.* despite what the man pages say. Why mount figures so much in this section will become clear later.

[The available tools for partitioning varies according to the version/revision of OS X Installer CD, and/or Restore CD.]

Tools for partitioning:

Although using the OSX/OSXS Restore CD would result in a quick setup of the system, the provided partitioning tool is Drive Setup, an OS 9.x application. Drive Setup is a very flexible application, trying to cater for classic MacOS, A/UX, and OSX. Unfortunately, not all options would be useful or valid for partitioning OSX/OSXS.

In Drive Setup, you can choose to partition the drive, and select the number of partitions (which will be equal in size). Once that's done, you can adjust the partition type and size. You can select the type of partition from some of the following:

The only partition types of interest to us would be HFS+ and UDF, since they're used by Mac OS 9.x and Mac OS X.

If you perform a clean install using the OSX/OSXS Installer, The Installer has a menu option for partitioning the disk using Disk Utility. Disk Utility is much cleaner, only providing the two partition types of interest. There's also an option to create software RAID partitions, but I haven't had any experience with it yet.

[Note: the following describes the procedure using Drive Setup, and is probably more involved that it should be.]

I wanted to use UDF as my experience with HFS+ has been somewhat ambivalent, when it comes to Darwin/command line support. In addition, UDF is a better match to the Unix roots that makes up Darwin/Mac OS X. At least one HFS+ partition is recommended in case you need to install MacOS 9.x (whether to boot from, or for Classic support). How many partitions? Well, I'm still not sure what the best setup is. However, for the 60 GB disk that came with our G4 Server, I did the following:

You are then prompted to format the drive. This is fine, except that I couldn't figure out how to format the UFS partitions (and I didn't know that Drive Setup didn't do so). This resulted in the Installer CD recognizing only the two HFS+ partitions upon exiting Drive Setup.

Formatting UFS Partitions

Once OS X was installed (I did a Restore from the CD), I have to figure out how to bring the 3 UFS partitions online from within OS X. This is where the fun with the mount command begins. [Rant: There is a big disconnect among the behavior of Darwin, OS X and the man pages in documenting the correct behavior of file system management commands (e.g., 'mount')]. I didn't expect mount to be able to mount unformatted partitions, but it's a place to start. The mount man pages threw me off track by referring to /etc/fstab, which exists as three separate files in OS X. Of course, that wasn't relevant anyway since /dev/hd0x didn't exist. Next, try mounting the physical disk via 'mount /dev/disk0sYY', with YY being the first number after the 10 GB HFS+ partition. mount returns an error indicating invalid superblocks. Ah.... at least it finds the partition. Now to format the UFS partitions.

OS X Disk Utility wasn't much of a help, as it doesn't have an option to format the already created UFS partitions (made more complicated by the fact that it's on the startup drive, which is locked). Man pages on newfs was mostly confusing, as there isn't a corresponding newfs_ufs for the UFS partition type. pdisk shows that the three partitions are marked as UFS, so I went ahead with just using newfs hoping that it'd figure out which type of parititon it is automatically. Fortunately, it did.

The newfs man page refers to disklabel to label the disk. Unfortunately, that is not part OS X. So I was stumped again trying to figure out how to map the disk to a mount point. Worse, when I finally managed to mount, the UFS partitions showed up in the shell listings for mount and df, but not in the Finder. [Rant: There is a really bad disconnect between OS X and Darwin here] It finally showed up in the finder after a reboot.

This is what I finally figured out (It may not be the fastest, or most optimum way -- this is from memory, I hope I got everything right):

  1. Setup needed partitions before installing OS X, with at least one HFS+ partition
  2. Boot into OS X
  3. You must be root, obviously
  4. use newfs /dev/disk0sYY, to format the relevant UFS partition.
  5. mount /dev/disk0sYY, (no additional arguments) for all UFS partitions (/etc/fstab is not used)
  6. Reboot
  7. From OS X Disk Utility, determine the device name /dev/disk0sYY and the corresponding label (for similarly sized partitions)
  8. In OS X Finder, change the label of the mounted volume(s) 'untitled X' to the desired volume name
  9. Migrate the various items (swap directory, /Users, etc). Note: Don't use cp (see section on Macintosh Files)
    [You can refer to http://www.xlr8yourmac.com/OSX/OS_x_swapfile_notes.html for information on moving the swap directory. However, I decided to follow the last method by Eugene Chan as it's supposed to be safer.]
  10. You should also take a look at Server Services for some caveats.

Macintosh Files

(with corrections/additions furnished by Thomas Kaiser)

[A good reference can be found in "Inside Mac OS X: System Overview,"
under "System Technologies->Darwin," and under "The File System"
http://developer.apple.com/techpubs/macosx/Essentials/SystemOverview/SystemTechnologies/Darwin.html
http://developer.apple.com/techpubs/macosx/Essentials/SystemOverview/FileSystem/index.html
A historical perspective can be found at http://www.mit.edu/people/wsanchez/papers/USENIX_2000/ under the "Filesystems" subtitle]

Traditionally, MacOS has always supported files containing a Data and a Resource Forks. This support was built into the HFS file system that was part of MacOS until MacOS 9.x. HFS+ was introduced later to address some of the limitations (file sizes, disk sizes, etc.) in anticipation of MacOS X. HFS/HFS+ also support metafile information such as file attributes describing the file type, owner (application) and other file status (locked, stationery, etc.) Transferring Macintosh files to non-Macintosh operating systems has always been somewhat troublesome. Apart from Carriage Return and Line Feed conversion for text files (between DOS, Unix and MacOS), we also have to deal with binary files that contain data and resource forks. Standard means of representing binary Macintosh files on other systems have been MacBinary and BinHex (ASCII-compliant) formats. In addition, Apple defines "Apple Single" and "Apple Double" formats (in the earlier A/UX days) to convert from HFS files to flat files that can be stored, transferred and retrieved correctly while preserving their meta-information.

MacOS X comes into the picture, combining its Unix roots with classic MacOS file systems. In a typical MacOS X installation, the file system contains HFS-based MacOS applications and files (Carbon, Classic), UFS friendly Quartz/Aqua Frameworks, applications and files (Cocoa), as well as traditional core POSIX (Unix) command line tools and files (Darwin).

As long as users access files via the Finder, all these issues are mostly hidden (I won't get into the MacOS X file types using extension mapping a-la MS-DOS issue here). However, life is not as simple for CLI geeks. MacOS X supports both HFS/HFS+ and UFS file systems for hard disks. UFS is a Unix file system that is filename case sensitive, and doesn't support Resource forks or HFS file attributes natively. Instead, Resource forks are stored as hidden files of the format "._filename". [How are HFS file attributes stored?] In addition, typical Unix file manipulation commands (e.g., cp, ls, rm) don't always work as expected due to their inability to handle HFS file attributes, regardless of the underlying file system.

Path Separators

In HFS+ and the finder, the colon ":" is used as path separators, and is not legal in Finder filenames, whereas forward slash "/" is allowed. On the other hand, UFS and command shells uses forward slash "/" as the path separator, and allows colon ":" to be used in filenames. Depending on whether you're accessing the file path via Finder (Carbon/Classic APIs) or command shell (Cocoa/Unix APIs), the presence of invalid characters ":" and "/" will be automatically mangled by OSX to become the other valid character. e.g., "Data/File" in HFS will become "Data:File" in UFS and command shells. This translation is done transparently and would not normally impact applications unless you're trying to cut and paste or exchange path names between command line and MacOS applications. However, to avoid ambiguity and confusion, it is best not to use either character in filenames.
[See http://developer.apple.com/techpubs/macosx/Darwin/General/KernelEnvironment/FilesystemFolder/index.html,
http://developer.apple.com/techpubs/macosx/Essentials/SystemOverview/SystemTechnologies/iDarwin.html,
http://developer.apple.com/techpubs/macosx/Essentials/SystemOverview/FileSystem/index.html,
http://developer.apple.com/techpubs/macosx/Essentials/SystemOverview/Finder/iThe_Finder___Operations.html for further details.]

File Permissions

[See file://localhost/Developer/Documentation/ReleaseNotes/Permissions.html for information regarding HFS+ privileges. Note: I'm summarizing the info here as is, I haven't studied the HFS+ privileges issue extensively]

File Permissions for UFS partitions are enforced using Unix mode flags, with extended attributes stored as metadata (in addition to the standard mode bits) to support HFS+ attributes. See the section on "Manipulating Carbon/Classic Files" on how to access the various attributes.

On HFS+ partitions, OSX treats such partitions as either Native (mounted locally), which enforces stored file permissions; or Foreign (mounted as network share), which allows for permissions to be overridden via Get Info->Privileges on the volume.

Issues relating to Apple File Sharing (AFP)

Caveat: The file manipulation actions (copy, delete, move) given in the next subsection are given in the context of local file mainpulation actions (i.e., on locally mounted partitions). It has not been verified to work correctly on Apple File Sharing (AFP) mounted volumes, especially with regards to UFS partitions, and has caused file/directory corruptions as of OSX 10.1.1.

Hence, the file manipulation actions would work correctly for AFP shares only if they're HFS+ partitions, as far as I am aware [Collaborating evidence/data requested, I suspect it is because the AFPFS daemon on OSXS is executing the system tools (ditto?) as root, and not with the client ID, nor do they take into account the client ID to change the resource fork ownership/permissions to the correct values].

[From Thomas Kaiser]
If you use the Finder on a client mac to copy a file from one of your server's volumes to another, it will only work, if they both use the same filesystem (please note: MacOS X Server supports the SupportsCopyFile Bit, which will result in less network traffic when you attempt to copy from one server volume to another --> the server will be asked to do the job for you, so that the file won't be transferred over the wire at all).

In the case you copy from one hfs volume to an ufs volume, the Finder will stop, saying "object is in use". On the UNIX side you can see an empty datafork, an empty ressource fork, the latter owned by 'root'! Looks like ditto is used on the server (see above). The only way to do so, is to copy the file from one server volume to your local harddisk and then back to the other volume.

Another restriction is the loss of Finder comments when copying files between volumes (when using the "SupportsCopyFile" trick)! Seems to be only a problem for afp clients since MacOS 10 stores Finder comments different for local use than for afp clients?

Manipulating Carbon/Classic Files

Warning: NEVER use POSIX tools (cp, mv, rm) on Carbon/Classic files residing on HFS+ partitions. They'll corrupt the resource forks and render the files unusable.

When dealing with UFS partitions, POSIX tools do not account for the separate resource fork files "._{filename}". Wrapper tools to take care of this situation should probably be developed and used in place of the default POSIX commands.

Manipulating Cocoa/Unix Files

Deleting 'Undeletable' Files

Due to the interaction between Unix permission flags and Apples HFS+ attributes, sometimes deleting seemingly normal files can be troublesome. A good summary from Apple's KBase is given below:

Archiving/Mirroring Files/ Disks

There are several tools that can be used to archive files on a Mac OS disk, as well as mirror it to another drive. However, all the OSX supplied tools have problems in dealing with Classic/Carbon files. There are third party tools available with various degree of usability and safety. Commercial tools are still in beta testing as of Feb 2002.

User Accounts

Server Services (OSXS)

[Note: Server Admin is severely broken in OSXS 10.0.x. So far, OSXS.1 seems to have a more stable Server Admin tool, but it's still not very easy to get things configured correctly]

Server Services are usually managed using the Server Admin tool. However, I find the tool in 10.0.4 to be still rather immature and frequently gives incorrect information or disconnects from the server unexpectedly. The network-based connection between the UI and the back end definitely needs improvement, especially when it comes to providing consistent status information after making a change.

Apple File Sharing

The Control Panel has a Sharing option, which isn't used to configure Share access. To make things more confusing, Get Info on a folder doesn't allow one to share the folder (I'm thinking like Windows now). As far as I can make it out, you can only configure sharing via the Server Admin tool under the General tab. There's two menu items, both of which need to be accessed to enable sharing on a given folder. Unfortunately, the two menus are exclusive, leading users to keep setting the sharing permissions, going to verify that it's shared, discovering that something's missing/not shared for strange reasons, return to sharing permissions to find that the earlier change didn't take....

Caveats: See "Issues relating to Apple File Sharing (AFP)" under Macintosh Files for warnings regarding sharing UFS partitions.

FTP (including Anonymous FTP)

Note: General consensus in security circles is that enabling FTP is not a good thing, since passwords are sent in the clear. Consequently, only anonymous FTP should be enabled. Users wishing to send and receive files using their accounts should use other means (e.g., AFP, scp, sftp) instead.

This is by far the most confusing and most frustrating service to configure in OS X if you have a non-standard configuration (which I did obviously after moving directories around to different partitions). First, there's two variants of ftp servers, the default ftpd, and wu-ftpd. I'm not sure why wu-ftpd is chosen in place of ftpd by the Server Admin tool in 10.0.4, possibly due to enabling anonymous FTP service? Of course, the man pages for wu-ftpd are missing. [From Apple's OS X Server discussion forum - http://www.wu-ftpd.org ] In any case, the man pages are incorrect - by default, the FTP configuration and data directories are in /Library/FTPServer/, not /etc/ftp*.

Sharing MUST be enabled on the directory that you wish to access via FTP. If it's not defined as a Share point, the directory is not accessible via FTP. Go figure.

Enabling Anonymous FTP properly is rather messy. There's an article on Mac OS X Hints that describes several ways of configuring it. I modified the version by David Schlosnagle as it was closest to my needs (download only). As mentioned earlier, ignore the /etc/ftp* files, use the versions in /Library/FTPServer/Configuration and /Library/FTPServer/Messages. Also, the ftp root directory ~ftp would be /Library/FTPServer/FTPRoot/.

Multiple Server Names after Reconfiguration

(Source: John DeTroye, MacOS-X-Server list)

If the server name and/or IP address is reconfigured, sometimes it causes clients to see multiple instances of the server. This is due to the File Sharing daemon SLP retaining obsolete information.You can try to fix this problem by deleting the slp.regfile on the SLP server (locate slp.regfile to find its location).

 

Updating Software and MacOS X Server Versions

The Software Update feature provides access to MacOS X updates. For OSXS, there is a two-step process to update the Server to a new version. The first is to apply the "Mac OS X Update 10.x.y" which is contains client code common to both OSX and OSXS, and then after a successful reboot, Software Update may prompt for a "10.x.y Server Update" (if available).

However, The "About This Macintosh" dialog box only indicates the Server Version number. The Client Version can only be determined via
Apple System Profiler, which will indicate both the Server and Client versions. This can be accessed via command line as AppleSystemProfiler to generate a report to stdout. (AppleSystemProfiler -h for details on possible output formats).

Tip: Click on the MacOS X Version line in the "About This Macintosh" dialog box to find out the MacOS X build number.

Note: As of OSX 10.1.2, The GUI based Apple System Profiler has a bug in the display window that hides the Client Version number. Use the CLI based AppleSystemProfiler to get the info.

Resources/ Sources

Acknowledgements

Feedback to <tcwan at cs dot usm dot my>.