Tuesday, March 20, 2007

Burning CDs from the GUI and command line

I recently downloaded the six CD images for CentOS 5 beta, and decided to try burning them to disks using an OS X machine. This turned out to be a little less straightforward than I would have expected, given OS X's usual attention to the user experience.

To burn an ISO image to a CD from the graphical environment, start up the finder and browse to the directory where the image resides. Right-click on the image file, then select "Open with..." --> "Disk Utility". This will open the disk utility window. Now click on the ISO image on the left-hand side of this window and then click the "Burn" button at the top left (the button with a menacing black-and-yellow radiation symbol on it). You'll be asked to confirm the automatic choice of burning device, and then Bob's Your Uncle.

You can accomplish the same thing from the command line, using the drutil command. For example:
drutil burn -eject filename.iso
Unless told otherwise, drutil will choose the first available CD burning device, and write the image to that. Command-line switches allow you to choose a specific burning device. The command
drutil list
will show you a list of attached devices.

Monday, November 27, 2006

Running Programs When a User Logs In

OS X provides a graphical tool (System Preferences --> Accounts --> Login Items) that users can use to cause selected applications to launch whenever the user logs in at the console. These settings are saved in the user's home directory, in the file ~/Library/Preferences/loginwindow.plist .

But what about system-wide settings that cause applications to launch for any user? It turns out that a user's loginwindow.plist file can just be copied into /Library/Preferences, and it will affect all users. If a user also has his or her own loginwindow.plist file, all applications in both the system-wide plist file and the user's plist file will be launched when the user logs in at the console.

Instead of editing the system-wide plist file, though, I've adopted the solution described by Greg Neagle in this Macenterprise.org article. Neagle creates a generic loginwindow.plist file that launches one application, called "LoginLauncher.app". This application then looks for files in a specified directory (these can be application bundles, executables, or AppleScripts) and launches each of them. Adding new items to be launched at login only requires dropping the application into the specified directory. This is similar to the MS Windows "Startup" folder.

LoginLauncher.app is an application bundle whose heart is a script called LoginLauncher, which looks like this:


echo "LoginLauncher: Running login items..."

# find the bundle contents dir
macosdir=`/usr/bin/dirname $0`
contentsdir=`/usr/bin/dirname $macosdir`

# use the defaults command to read in the LoginItemsDir from
# $contentsdir/Resources/Defaults.plist
loginItemsDir=`/usr/bin/defaults read "$contentsdir/Resources/Defaults" LoginItemsDir`
echo "LoginLauncher: loginItemsDir is $loginItemsDir"

for file in "$loginItemsDir"/*
echo "LoginLauncher: Processing $file..."
if [ -x "$file" -a ! -d "$file" ]; then
# execute it
echo "LoginLauncher: Executing: $file"
"$file" &
echo "LoginLauncher: Not an executable file..."
macName=`osascript -e "get POSIX file \"$file\" as text"`
if [ $? -eq 0 ]; then
kind=`/usr/bin/osascript -e "tell application \"Finder\" to get kind of item \"$macName\""`
if [ "$kind" == "Alias" ]; then
kind=`/usr/bin/osascript -e "tell application \"Finder\" to get kind of original item of item \"$macName\""`
if [ "$kind" == "Script" -o "$kind" == "script text" -o "$kind" == "compiled script" ]; then
# run the Applescript
echo "LoginLauncher: Running Applescript: $file"
/usr/bin/osascript -e "tell application \"Finder\" to run script file \"$macName\""
# just pass it to the open command, which will launch apps and open docs
echo "LoginLauncher: Opening: $file"
/usr/bin/open "$file"

echo "LoginLauncher: Completed running login items."
(This is a slightly modified version of Greg Neagle's script.)

The script first gathers a list of the contents of the login items directory (specified
in a plist configuration file in the LoginLauncher application bundle). For each item in the list, the script then checks to see if the item is an executable file and, if so, executes it. If the item isn't an executable file, the script uses osascript to execute a snippet of code that determines the file's type, then does something appropriate based on that determination.

The AppleScript statement "get POSIX file \"$file\" as text" uses something from the AppleScript "Standard Additions", but I can't find any comprehensive documentation for the Standard Additions online.

Wednesday, November 15, 2006

Quartz, Classic/Carbon/Cocoa, and Aqua

Names like "Quartz", "Carbon" and "Aqua" get thrown around a lot when people talk about OS X, and it takes some digging to figure out what they mean. (Especially since they're often used loosely or even inaccurately.) In this post, I'll try to disentangle the semantic mess and draw parallels between the OS X graphics system and the X graphics system used on most Linux desktops.

I'll start with a diagram showing the two graphics stacks side-by-side:

 OS XLinux
Window ManagerAquatwm,mwm,kwin,sawfish,enlightenment,fvwm,etc...
ToolkitsClassic, Carbon, CocoaMotif, Qt, gtk, etc...
Display ServerQuartzX11
Non-graphical CoreDarwinLinux kernel + libs + non-graphical programs

Quartz sits at the base of the graphical stack, corresponding to the role of the X server on a Linux desktop, where it handles the drawing of graphics primitives on the display. Quartz is the descendant of the Display Postscript system used in the NeXT operating system. Instead of Postscript, though, Quartz internally represents graphical elements as PDF. Graphics entities are represented inside Quartz as high-level, scalable abstractions which can be zoomed, rotated, stretched, drop-shadowed or otherwise manipulated dynamically as they are rendered to the screen. Applications built on top of Quartz inherit these capabilities for free. This is very different from the X server world, where primitives are drawn as raster images, and any more sophisticated manipulations have to be done at higher levels in the graphics stack.

Sitting on top of Quartz are several APIs. The "Classic" API allows backward-compatibility with OS 9 applications. The "Carbon" API is a new interface allowing programmers to take better advantage of the capabilities of Quartz. "Cocoa" is an object-oriented API, also developed for use with Quartz. Classic/Carbon/Cocoa correspond roughly with the various tookits used for writing X applications.

Finally, Aqua is a user interface sitting at the top of the stack. It implements window decorations and allows the user to manipulate entities on the graphical display through the keyboard and mouse. It corresponds to the window manager in a typical Linux desktop configuration.

For much more information, see this excellent article by John Siracusa at Ars Technica.

Tuesday, November 14, 2006

Setting the DNS Domain Search Path

Although OS X has many features in common with Linux, sometimes looks can be deceiving. For example, under Linux I would set the DNS domain search path by editing /etc/resolv.conf and adding a line like
search department.company.com company.com
which would cause the system to search first for "host.department.company.com" then, if that fails, "host.company.com". Under OS X, you'll find that there are two "resolv.conf" files:
# locate resolv.conf
but neither of them can actually be used to do what I described above, although the contents of the files looks like a regular resolv.conf file under Linux.

Since OS X 10.4, these resolv.conf files are just for informational purposes, and even though you can modify the files, many applications will ignore them. Fortunately, there's a convenient command-line tool for changing the underlying DNS settings. The networksetup command can do it for you, like this:
 /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Support/networksetup \
-setsearchdomains 'Built-in Ethernet' department.company.com company.com

According to this post it also seems to be possible to use scutil to do this, although I haven't tried it and it looks more complicated.

Monday, November 13, 2006

A Firewall Script

In an earlier post I talked about Apple's simple firewall controls. These are just a front-end for the BSD-derived ipfw utility. To have more control over the firewall configuration, I'm now using a script (run as a StartupItem) to configure the firewall. I stole the script and associated configuration file from this site at the University of Washington, then modified the script slightly. The end result follows:

# Based on work done by Stefan Arentz:
# http://wopr.norad.org/articles/firewall/
# And other random information sources, e.g. ipfw(8), "Building
# Internet Firewalls" by O'Reilly, and random sample ipfw scripts.


# attempt to dynamically set the interface to use
# in case Apple changes the driver name at some point
# (everything I have seen thus far has been en#)
#INTERFACE=`ifconfig -a | egrep -i '^[A-Za-z0-9]+: ' | grep -v lo | cut -f 1 -d :`

#MYADDR=`ifconfig $INTERFACE | grep inet | awk '{print $2}'`


# required startup script statements
. /etc/rc.common
ConsoleMessage "Configuring Firewall"

# for diagnostics

# Turn on logging of packets for debugging/auditing.
if [ `$SYSCTL -n net.inet.ip.fw.verbose` == 0 ] ; then
$SYSCTL -w net.inet.ip.fw.verbose=1

# limit is how many packets will be logged for a specific rule
# good for avoiding fill-the-disk logging attacks
if [ `$SYSCTL -n net.inet.ip.fw.verbose_limit` != 1000 ]; then
$SYSCTL -w net.inet.ip.fw.verbose_limit=1000


# JAM 2001-10-02 new 10.1 bug (?) stalls boot when starting up not
# connected to any network, with system.log entries of:
# mach_kernel: ipfw_load
# mach_kernel: IP packet filtering initialized, divert enabled,
# rule-based forwarding enabled, default to accept, logging disabled
# /sbin/ipfw: NetInfo timeout connecting to local domain, sleeping
# therefore, wrap script as background subshell until figure out why
# stall is taking place

# flush all the rules
$IPFW -f flush

# Local loopback interface is open
$IPFW add allow ip from any to any via lo0
# however, we otherwise ban loopback address traffic
$IPFW add deny all from any to
$IPFW add deny all from to any

# ip-options (per FreeBSD Security Advisory: FreeBSD-SA-00:23.ip-options)
$IPFW add deny log ip from any to any ipoptions ssrr,lsrr,ts,rr

# Allow TCP through if setup succeeded
$IPFW add pass tcp from any to any established

# Allow IP fragments to pass through
#$IPFW add pass log all from any to any frag


# allow DHCP & DNS
$IPFW add allow udp from any 67 to any 68 in via $INTERFACE
$IPFW add allow udp from any 53 to any in via $INTERFACE

# send RST back on auth to timeout remote requests faster
$IPFW add reset tcp from any to any 113 in via $INTERFACE setup

# allow NTP server responses
$IPFW add allow udp from any 123 to any in via $INTERFACE
$IPFW add allow udp from any 123 to any in via $INTERFACE

# let SSH, WWW
$IPFW add allow tcp from any to any 22,80 in via $INTERFACE setup

# allow AFP services to work (ICMP ping also required)
$IPFW add allow tcp from any to any 427 in via $INTERFACE setup
$IPFW add allow udp from any to any 427 in via $INTERFACE
$IPFW add allow tcp from any to any 548 in via $INTERFACE setup
$IPFW add allow udp from any to any 548 in via $INTERFACE

# Allow Zebedee
$IPFW add allow tcp from any to any 11965 in via $INTERFACE setup

# block all other incoming TCP/UDP traffic
$IPFW add 65435 reset tcp from any to any in via $INTERFACE
$IPFW add 65435 unreach port udp from any to any in via $INTERFACE


# allow certain ICMP through (allows ping, traceroute, plus
# the required source quence and similar)
$IPFW add allow icmp from any to any icmptypes 0,3,4,8,11,12

# silent block on router advertisements
$IPFW add deny icmp from any to any icmptypes 9

# drop all other ICMP
$IPFW add 65435 deny icmp from any to any

} &

I've added in a rule allowing zebedee (which I use for tunneling VNC traffic) and I've hard-coded the interface name as "en0", since under OS X 10.4 on my test machine the original code returns a long list of interfaces, rather than a single interface name:

It would be useful to modify the script so that it sets the same firewall rules for all of these interfaces.

Friday, November 10, 2006

Groups, Administrators

As you can see from the adduser script in a previous post, users can be added to a group with a command like
niutil -appendprop / /groups/$groupname users $username
Similarly, a user can be removed from a group with a command like
niutil -destroyval / /groups/$groupname users $username
In my configuration, I've decided to follow Red Hat's convention of having a group for each user, and making that the user's primary group.

Other groups of interest include the admin group, and the appserveradm and appserverusr groups. The admin group is important because it's (by default) listed in the /etc/sudoers file. Out of the box, this file contains only the lines:
root    ALL=(ALL) ALL
%admin ALL=(ALL) ALL
(except for comments). The second line allows anyone who's a member of the admin group to use sudo to run any command as the superuser.

The second two groups contain users who are allowed to manage WebObjects, a Java web application server thingy. By default, the first user you create while setting up a new Mac will be made a member of these two groups.

In my case, I've chosen to allow root to log in directly and remove all other users from the admin group. To accomplish this, just log in as a still-privileged user and type "sudo passwd root" to set a password for the root account. Then use the "niutil -destroyval" command above to remove users from the admin group (after testing to make sure you can log in as root!).

Installing Geant4, and a fink/Apple dylib name conflict

Geant4 is a particle physics simulation toolkit. The easiest way I found to install it under OS X was to first fetch the latest source code then follow steps similar to those described for installing the SLAC Geant4 tutorial material. The key to this is creating an appropriate "setup-geant4.sh" script that will set the values of several environment variables. My first attempt at this is given below:


export G4HERE=`pwd`
export G4INSTALL=$G4HERE/geant4.8.1.p01

# Check we are where we think we are
if ! test -e $G4INSTALL/source/GNUmakefile
echo "G4INSTALL not set correctly. Please edit setup-geant4 script to fix problem"

export G4SYSTEM=Darwin-g++
export CLHEP_BASE_DIR=/sw
if ! test -e $CLHEP_BASE_DIR/include/CLHEP/config/CLHEP.h
echo "CLHEP_BASE_DIR not set correctly. Please edit setup-geant4 script to fix problem"

if [ -z "$PATH" ] ; then
export PATH=""
export PATH=${PATH}:${G4INSTALL}/bin/${G4SYSTEM}

# If we run WIN32-VC* , it also means that it is cygwin
case "$G4SYSTEM" in
export CLHEP_BASE_DIR=`/bin/cygpath -m "$CLHEP_BASE_DIR"`
export G4INSTALL=`/bin/cygpath -m "$G4INSTALL"`
if [ -z "$LIB" ] ; then
export LIB=""
if [ -z "$lib" ] ; then
export lib=$LIB
export lib="${CLHEP_BASE_DIR};$lib"
# MacOS X
if [ -z "$DYLD_LIBRARY_PATH" ] ; then
# default
if [ -z "$LD_LIBRARY_PATH" ] ; then

export G4LIB_USE_ZLIB=1


I put this script into /usr/src, then fetched geant4.8.1.p01.gtar.gz and unpacked it into /usr/src/geant4.8.1.p01 . I then typed "cd /usr/src; . setup-geant4.sh" to set the appropriate environment variables. Then I followed the steps outlined in the SLAC documentation and typed "cd geant4.8.1.p01/source; make". After make completed, I tried one of the example programs, following the steps outlined for example A01 in the SLAC tutorial. The program compiled, but I found that it wouldn't produce graphics on an X display. To solve this, I needed to set two more environment variables:


I then went back to the geant4 source directory, did a "make clean" and another "make". Once this had completed, I re-made example A01 and found a different problem. Now I was getting a run-time error that said:

dyld: Symbol not found: __cg_jpeg_resync_to_restart
Referenced from: /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ImageIO.framework/Versions/A/ImageIO
Expected in: /sw/lib/libJPEG.dylib

Trace/BPT trap

It turns out that Apple ships a "libJPEG.dylib" with OS X which is completely different from the "libjpeg.dylib" of the same name (modulo case) that fink provides. As you can see above, the setup-geant4.sh script I copied from SLAC adds the directory /sw/lib (where the fink-installed libjpeg.dylib lives) to DYLD_LIBRARY_PATH. The Geant4 program was looking for the Apple-supplied library, but because DYLD_LIBRARY_PATH told it to look in /sw/lib first, it found the fink-supplied file of the "same" name instead. Removing /sw/lib from DYLD_LIBRARY_PATH fixed the problem, and the example ran as expected thereafter.

Note that there is a similar potential name conflict between libGIF.dylib / libgif.dylib. These problems have been noticed by others.

Finally, to set up the environment for compiling Geant4 programs in the future, I copied /usr/src/geant4.8.1.p01 into the master copy of /common/lib/geant4.8.1.p01 and made a symlink from there to /common/lib/geant4. I then arranged to have the following environment variables defined:


Wednesday, November 08, 2006

An adduser Script

Although OS X provides a graphical interface for adding users, there's no simple command-line utility for doing it. So, I've taken the adduser script I use for the Linux computers I manage, and modified it to work under OS X.

OS X stores user account information in a NetInfo heirarchical database. Information can be modified, added to or extracted from the database through commands like "nireport" and "niutil". For example, to get a list of all user accounts, you could type:
nireport / /users name

Here's the adduser script I've come up with. This is influenced by examples found on the web, but I'm especially indebted to Dave Dribin for his clear, detailed description of shadow files under OS X. I've left out a central section of the following script, in which I attempt to look up the user in our institution's central database. I omitted this both for security-through-obscurity reasons, and also because it probably wouldn't be applicable elsewhere. Insert your own code here if you need it. After the code listing, I'll describe some features of the script.

use strict;
use Digest::SHA1 qw/sha1_hex/;

$ENV{'PATH'} = "/usr/sbin:/usr/bin:/bin";

my $userid;
my @userlist;
if ( $ARGV[0] ) {
chomp( $userid = shift() );
if ( "$userid" eq "list" ) {
print "Enter file name: ";
chop(my $userfile = <>);
open (USERFILE,"<$userfile") or die "Can't open $userfile: $!\n";
chomp(@userlist = <USERFILE>);
close (USERFILE);
} else {
push (@userlist,$userid);
} else {
print ( "Enter userid: " );
chop($userid = <>);
push (@userlist,$userid);

USERID: for my $userid (@userlist) {

if ( getpwnam($userid) ) {
print "$userid already has an account.\n";
mkhome ($userid);
next USERID;

# Look user up in LDAP database:
# ...insert appropriate magic here for your institution, if needed.

if ( "$name" ne "" ) {
print "Found $name in LDAP database. OK? (y|n) [n]: \n";
chop(my $ans = <>);
if ( "$ans" ne "y" && "$ans" ne "Y" ) {
$name = "";
if ( "$name" eq "" ) {
print ( "Enter user name: " );
chop($name = <>);

print ( "Creating new account for [$userid]. . .\n" );

# Find first unused uid:
chomp ( my $nextuid = `nireport / /users name uid | awk '{print \$2}' | sort -n|tail -1` );
# Find first unused gid:
chomp ( my $nextgid = `nireport / /groups name gid | awk '{print \$2}' | sort -n|tail -1` );

# Create NetInfo data:
`niutil -create / /users/$userid`;
`niutil -createprop / /users/$userid uid $nextuid`;
`niutil -createprop / /users/$userid realname "$name"`;
`niutil -createprop / /users/$userid home "/Users/$userid"`;
`niutil -createprop / /users/$userid shell "/bin/bash"`;
`niutil -createprop / /users/$userid gid $nextgid`;
`niutil -createprop / /users/$userid passwd "********"`;
`niutil -createprop / /users/$userid authentication_authority ";ShadowHash;"`;

# Generate a 'generateduid':
chomp( my $generateduid = `uuidgen` );
`niutil -createprop / /users/$userid generateduid "$generateduid"`;

`niutil -create / /groups/$userid`;
`niutil -createprop / /groups/$userid gid $nextgid`;
`niutil -appendprop / /groups/$userid users $userid`;

# Generate a 'generatedgid':
chomp( my $generatedgid = `uuidgen` );
`niutil -createprop / /groups/$userid generatedgid "$generatedgid"`;

# Make suggested password:
chomp( my $pwsugg = `/common/bin/pwsuggest` );
my $password;
print ( "Enter password [default $pwsugg]: " );
chop ($password = <>);
if ( "x$password" eq "x" ) {
$password = $pwsugg;

# Create password hash:
my $salt1 = chr(int(rand(26))+64);
my $salt2 = chr(int(rand(26))+64);
my $salt3 = chr(int(rand(26))+64);
my $salt4 = chr(int(rand(26))+64);
my $salt = $salt1.$salt2.$salt3.$salt4;
my $hex_salt = $salt;
$hex_salt =~ s/(.)/uc(sprintf("%2.2x",ord($1)))/eg;
my $cpw = sha1_hex($salt.$password);

my @data = ();
for (my $i=0;$i<1240;$i++) {
$data[$i] = 0;
@data[168..215] = split(//,$hex_salt.$cpw);

# Store password hash:
open ( HASHFILE, "> /var/db/shadow/hash/$generateduid" ) or die "Can't open hash file for generateduid=$generateduid: $!\n";
for (my $i=0;$i<1240;$i++) {
print HASHFILE uc($data[$i]);
close (HASHFILE);

mkhome ($userid);


sub mkhome () {
my $userid = shift;

my ($name,$passwd,$uid,$gid,
$quota,$comment,$gcos,$home,$shell,$expire) =
unless ( -d "$home" ) {
print "Making home directory $home...\n";
`cp -R "/System/Library/User\ Template/English.lproj" $home`;
`chown -R $userid:$userid $home`;

Some things to note about the script:

  • Even though user information is stored in a NetInfo database, the perl functions like "getpwnam" still work as expected.

  • You can use commands like "nireport / /users name uid" to fetch lists of selected properties (name and uid, in this case) from the NetInfo database. Also, although it isn't used in the script above, you can dump the NetInfo user data in plain old "passwd" format by using the command "nidump passwd ." (note the trailing dot). Similarly, "nidump group ." works.

  • The script creates a user in the NetInfo hierarchy by using "niutil -create" to create a new node for the given user, and then uses "niutil -createprop" to fill in the properties of that node. These should be self-explanatory, except for some subtleties in the way "passwd" works and the (to me) unfamiliar properties "authentication_authority" and "generateduid".

  • Beginning with recent versions of OS X, password hashes are stored using a shadow mechanism which makes the hashes unavailable to unprivileged users. Password hashes can still be stored in directly in the "passwd" property of a user's NetInfo entry, but this is not as secure. The "********" entry in the passwd property and the ";ShadowHash;" value of authentication_authority indicate that the accounts the script creates are using shadow passwords.

  • Under OS X shadow passwords are stored in files under /var/db/shadow/hash. Each file is named after a user's "generateduid". The generateduid is a "UUID" (Universally Unique Identifier) associated with each account. As in the script above, uuids can be generated by the command "uuidgen". The script assigns a uuid as each user's generateduid, and then creates the appropriate file in the /var/db/shadow/hash directory.

  • As described in Dave Dribin's article, the format of the shadow files has changed in OS X 10.4. The script above writes shadow files in the new format, so it probably won't work with earlier versions of OS X.

  • Finally, on Linux computers I use "mkpasswd" to generate random initial passwords. There doesn't seem to be anything similar packaged with OS X, so I just copied mkpasswd (which is just an expect script) and renamed it to "pwsuggest" for clarity.

CERNLIB, ddd, case conflict, ppc binaries, DYLD_LIBRARY_PATH

While installing CERNLIB I ran into problems with the currently-available g77 and gfortran for Intel OS X. CERNLIB is a set of libraries and utilities including PAW and PAW++, tools for analyzing data from particle physics experiments. CERNLIB makes heavy use of special features of g77, and although it can be compiled with gfortran on Intel Macs (after applying Keisuke Fujii's patches), the resulting binaries have run-time errors that make them unusable. (CERNLIB developers are aware of this problem.) The unofficial g77 port to Intel OS X can't build working binaries either.

The runtime errors are thrown by CERNLIB's ZEBRA system for managing dynamic data structures under Fortran 77 (!). For reference, the error messages look something like this:

!!!!! ZFATAL called from MZIOCH
called from MZFORM

!!!!! ZFATAL reached from MZIOCH for Case= 1

IQUEST(11) = ********* 52494448 HDIR
IQUEST(12) = 0 0
IQUEST(13) = 5 5

In an attempt to isolate theproblem, I installed the excellent debugger, DDD. I installed from source, putting the executable into my /local directory tree.

While installing DDD, I ran into a conflict between the "case philosophies" of Linux and OS X. OS X is "case preserving", meaning that it will display the case of characters in file names, but internally it doesn't care whether the file is named "JoEBob.dAt" or "joebob.dat". Linux is truly "case sensitive", and distinguishes between files whose names only differ in case.

Some X applications will make use of application-specific X resources files (called app-defaults files). These files are typically named after the X application's "class", which is usually the same as the application name, but capitalized. While installing DDD, I found that the newly-created binary ("ddd") was overwriting the app-defaults file ("Ddd") which was stored in the same directory. I fixed the problem by moving the binary and re-making the app-defaults file.

Watching an errant CERNLIB program run with DDD led me deep into the guts of ZEBRA, but didn't provide any obvious solution to the problem. So I decided to fall back to the pre-compiled PowerPC binaries for CERNLIB. These are available in the Fink repository as "cernlib2005*.deb". In order to install these with dpkg, I had to append the flag "--force-architecture", since the debs were compiled for PowerPC rather than Intel.

After installing the package, an attempt to run pawX11 resulted in:

# pawX11
dyld: Library not loaded: /usr/X11R6/lib/libX11.6.dylib
Referenced from: /sw/bin/pawX11
Reason: no suitable image found. Did find:
/usr/X11R6/lib/libX11.6.dylib: mach-o, but wrong architecture
Trace/BPT trap

This was because the installed X11 libraries (from fink's xorg packages) were for Intel. To get around this, I fetched the PowerPC version of the xorg-shlibs deb. Rather than installing the package, I then extracted the contents of this deb into a directory, using "dpkg-deb --extract", then put the files into a convenient location ("/common/lib/ppc") that could be shared among all of the Macs I'll be managing. Finally, I set the DYLD_LIBRARY_PATH environment variable to point to this directory. This causes OS X's dynamic linker ("dyld", which is equivalent to Linux's "ld.so") to search first for dynamic libraries in /common/lib/ppc, before it goes on to search its default path. After these changes, pawX11 runs properly. In order to make paw++ work, I had to fetch the openmotif3-shlibs package for PowerPC and extract its contents into the same directory.

Friday, July 14, 2006

fink, apt and dpkg

Under OS X there doesn't appear to be any official (or de facto standard) full-featured package-management system for third-party software. Instead, there's the minimal "installer" command supplied by Apple, and several third-party alternatives. Installer lacks several desirable features:
  • It's not aware of dependencies
  • It provides no mechanism for un-installing packages
  • It provides no easy way of finding information about installed packages (although it does maintain such metadata in /Library/Receipts)
Among the alternatives to installer, one that stands out is the "fink" project.

The fink project consists of
  1. A port of the Debian "apt" and "dpkg" package-management tools to OS X
  2. A set of tools for automating the creation of binary packages in .deb format from source code
  3. A set of repositories of packages available for OS X
In the following I'll talk about installing fink, using a few of the commands it provides, and integrating fink into our general-purpose nightly update system.

The fink project provides a binary installer for PowerPC- and Intel-based OS X, or you can install from source code. I opted for the latter. The source code can be downloaded from the fink project source download page. In order to compile it (and in order to later use fink to build binary packages) you'll need to have Apple's Xcode development environment installed. By default, fink will be installed in a new directory tree called "/sw". This tree will contain the fink binaries (fink, apt, dpkg, etc.), configuration files, and most (but not quite all) of the software you install using fink (the notable exception is X11, which gets installed directly under /usr.). Executables installed through fink will (again , mostly but not quite always) be put into /sw/bin, so you'll need to add this to your search path. Some tools used to build and install packages will live in /sw/sbin, so this should be in root's search path.

The Debian apt and dpkg commands are analogous to yum and rpm in the Red Hat/Fedora world I'm more familiar with. Dpkg is a tool for installing, removing and reporting information about individual packages, and apt provides a framework for automatically resolving dependencies and fetching packages from repositories. The fink command provides an additional layer, capable of fetching source code from repositories and creating binary packages from it.

Here are a few useful dpkg commands:
  • dpkg -l will list all of the currently-installed packages
  • dpkg -L package will list the files installed by package
  • dpkg -s package will show meta-information about an installed package
  • dpkg -i package.deb will install package.deb
  • dpkg -r package will uninstall package

The most useful fink command for our purposes is "fink build package", which will create a binary .deb package from source code fetched from a repository. For example, the command "fink build tree" will fetch the source code for the useful "tree" command, then compile it and pack into a .deb file under /sw/fink/dists/stable/main/binary-darwin-i386/utils . This binary package could then be installed (on this or any other machine) with "dpkg -i".

Using only these tools, we can now add fink-produced .deb packages to our local package repository, along with dmg, app and pkg packages. To do this, I just wrote a trivial script called "deb-install" that looks like this:
exec dpkg -i $*
This will automatically be invoked when our pkgupdate script encounters a .deb file.

Notice that I've completely skipped apt for now. By doing this, I lose a couple of things:
  • Automatic dependency resolution. I have to manually sort out the packages, and make sure dependencies are satisfied
  • Automatic updates. I need to keep an eye out for updated packages, and add them to my repository
I think we can live without these for a while. The positive points of this solution:
  • I can keep all packages in one place
  • I can manage the complete list of packages (of all kinds) from one place
  • I have complete control over what is and isn't installed
Ultimately, I'll probably move apt into the process, by creating my own apt repository (or, if possible, making the current "universal" repostory apt-compatible), but I'll probably always want to check updated packages before I release them onto production machines.

I've now added many .deb packages to the repository and /common/manager/update.pkglist, and things seem to be working fine.

Tuesday, July 11, 2006

A General-Purpose Nightly Update Script

This is the fourth (and last) in a series of posts about a simple nightly update system for OS X.

In previous posts I've described Apple's "softwareupdate" command, which can be used to keep Apple-supplied software up-to-date, and I've described a set of scripts that can be used to keep third-party packages (at least if they're packaged in certain formats) up-to-date. These are necessary components of a nightly update system, but we still need more.

We'd also like to accomodate software that isn't packaged. For example, scripts we've written locally, or software that we've installed from source. For this purpose I've traditionally kept a couple of local directory trees that are synchronized nightly with master copies living on a server.

In addition, we'd like to keep certain local configuration files in sync with master copies. For example, we might want to com.apple.loginwindow.plist the same on all machines, and be able to change this file on all machines from time to time.

But there are other tasks besides software updates and configuration file updates. To accomodate these, we'd like the ability to run one or more catchall scripts on a nightly basis. The scripts should be syncronized with master copies living on a server, and can be used to perform whatever tasks we find necessary on a given night.

To do all of this I've written the following script, called just "update", which is based on an analogous script we've been using for many years to maintain our Linux computers.

Script update

# This script is run daily to update OS X computers running the standard
# configuration.

use strict;

$PkgUpdate::noyaml = 0;
unless (eval "use YAML; 1") {
warn "YAML not found, falling back to simple config.\n";
$PkgUpdate::noyaml = 1;

use File::Temp qw/tempdir/;

my $configfile = "/etc/localconfig";
my $window = 4*60*60; # 4 hours

# Establish lock on update system:
if ( -f "/var/run/update" ) {
chomp( my $OLDPID = `cat /var/run/update` );
chomp( my $PROCESS = `ps -p $OLDPID |wc -l` );
if ( $PROCESS > 1 ) {
`echo "Update system is locked!" | /usr/bin/mail -s "Message from Update" root`;
} else {
print "Removing stale lock file.\n";
unlink ("/var/run/update");
`echo $$ > /var/run/update`;

# Unless instant gratification is requested, wait for a random
# time, from 0 to 4 hours, to avoid hitting the server too hard
# all at once:
unless ( $ARGV[0] eq "now" ) {
my $wait = int(rand($window));
print "Sleeping for $wait seconds...\n";

# Load local configuration:
my $config;
my %simpleconfig;
unless ($PkgUpdate::noyaml) {
$config = LoadFile("$configfile.yml") or die "Cannot load config file $configfile.yml: $!\n";
} else {
%simpleconfig = simpleconfig($configfile);
$config = \%simpleconfig;

# Resync /common/manager:
if ( $config->{CACHE_COMMON} eq "yes" or
$config->{CACHE_COMMON_MANAGER} = "yes" ) {
print "Resynchronizing /common/manager...\n";
`chflags -R nouchg /common/manager`;
print `rsync -aqz --delete $config->{UPDATEMASTER}::update-$config->{UPDATEVERSION}/common/manager/ /common/manager/`;
`chflags -R uchg /common/manager`;

# Run pre-update script:
if ( -x "/common/manager/update.prescript" ) {
print "Running pre-update script...\n";
print `/common/manager/update.prescript`;

# Resync config files with master copies:
print "Resynchronizing config files...\n";
chomp( my $NOW = `date +%y%m%d%H%M%S` );
my $EXCL = "";
my $INCL = "";
-f "/etc/CACHE_CONFIG.exclude" && ($EXCL="--exclude-from=/etc/CACHE_CONFIG.exclude");
-f "/etc/CACHE_CONFIG.include" && ($INCL="--include-from=/etc/CACHE_CONFIG.include");
-d "/etc/oldconfigs" || mkdir("/etc/oldconfigs");
mkdir ("/etc/oldconfigs/$NOW");
print `rsync -aqz -b --backup-dir="/etc/oldconfigs/$NOW" -u $EXCL $INCL $config->{UPDATEMASTER}::update-$config->{UPDATEVERSION}/config/ /`;
# If oldconfig directory is empty, remove it:
rmdir ("/etc/oldconfigs/$NOW");

# Resync with master images, if requested:
if ( $config->{CACHE_COMMON} eq "yes" ) {
print "Resynchronizing /common...\n";
`chflags -R nouchg /common`;
print `rsync -aqz --delete $config->{UPDATEMASTER}::update-$config->{UPDATEVERSION}/common/ /common/`;
`chflags -R uchg /common`;
if ( $config->{CACHE_LOCAL} eq "yes" ) {
print "Resynchronizing /local...\n";
my $excl = "";
my $incl = "";
-f "/local/etc/CACHE_LOCAL.exclude" && ($excl = "--exclude-from=/local/etc/CACHE_LOCAL.exclude");
-f "/local/etc/CACHE_LOCAL.include" && ($incl = "--include-from=/local/etc/CACHE_LOCAL.include");
print `rsync -aqz -u $excl $incl $config->{UPDATEMASTER}::update-$config->{UPDATEVERSION}/local/ /local/`;

# Do Apple softwareupdate:
print "Doing Apple Softwareupdate...\n";
print `softwareupdate --install --all`;

# Do local pkg updates:
print "Doing Local Package Updates...\n";
print `/common/manager/pkgupdate /common/manager/update.pkglist`;

# Run post-update script:
if ( -x "/common/manager/update.postscript" ) {
print "Running post-update script...\n";
print `/common/manager/update.postscript`;

# Print time stamp and unlock update system:
my $now = localtime();
print "Update completed at $now\n";
unlink ("/var/run/update");

sub simpleconfig {
my $file = shift;

my %config = ();
open (FILE,"<$file") or die "Cannot open simpleconfig file $file: $!\n";
while (<FILE>) {
/^\#/ && next;
$config{$1} = $2;
close (FILE);
return %config;
The local directory trees mentioned above are /common and /local. There are two of them for a couple of reasons: one historical and probably no longer valid, and another still relevant. The historical reason is that Once Upon A Time we couldn't count on the local machine having enough disk space to accommodate all of the files we wanted to put into local directory trees, so we split the original "/common" and put some of the files (the ones that would be accessed most often) into "/local". We could then NFS-mount /common on machines with small disks.

This arrangement proved useful for another reason, even after disk sizes far outstripped our needs. It allowed us to distribute management responsibilities, where appropriate. For example, a user's search path now looks something like this:
If a computer has a local administrator, he or she can effectively replace any software in /common/bin by installing a file with the same name in /local/bin. More about this later.

The "update" script begins by checking to see if any older update processes are still running. In the past, we've sometimes seen network problems that cause update processes to pile up on a machine. This locking step was implemented to avoid that problem. If a valid lock is found, the process e-mails the local "root" account (which is just an alias in /etc/aliases, pointing to a central e-mail address for receiving such reports) and exits. Otherwise, the update script establishes its own lock and proceeds.

At this point, the script normally waits for a random time between 0 and 4 hours in order to spread out the load on the servers that supply update information. This wait can be skipped by supplying a command-line flag ("now").

The update script itself, and other associated scripts, live in the directory /common/manager. Before updating anything else, the update script first resynchronizes this directory with a master copy living on a central server. This ensures that changes to any of the update configuration files or subsidiary scripts will be made before the rest of the update procedure is executed. (Note that any changes to the update script itself won't take effect until after two update cycles. This could be avoided by re-exec'ing /common/manager/update with an appropriate flag after /common/manager has been updated, but we've never found that necessary.)

Between updates, all flags in the /common filesystem are kept in an immutable state. This prevents accidental (or trivial malicious) modification of these files, and it reinforces the idea that /common should always be an exact copy of a master image stored on a central server. Any local changes should be made in /local instead. Before synchronizing /common/master (or, later in the script, /common itself) the immutable flag is temporarily lifted, and then restored after the synchronization is complete. Note that under OS X we're using the "uchg" flag instead of "schg", since we need to turn immutablity on and off.

The actual syncronization is done with rsync. For /common/manager, the script constructs an rsync source address of the form:
and synchronizes that with the destination /common/master/ . The parameters UPDATEMASTER and UPDATEVERSION are drawn from a local configuration file, either /etc/localconfig or /etc/localconfig.yml.

Local configuration flags, CACHE_COMMON and CACHE_COMMON_MASTER allow a local administrator to turn off synchronization of all of /common or only /common/master.

After /common/manager is updated, the script looks for an executable file called /common/manager/update.prescript. If this is found, it is executed. The update.prescript file can contain any commands deemed necessary to be done before the rest of the update process. Later on, after most of the update process is completed, the update script will also look for an update.postscript script. As a matter of local convention, we make additions to these scripts cumulative. In other words, whenever we add a task to one of the pre- or post-scripts, we add code to check to see if the task has already been done, and skip it if so. This allows us to bring up-to-date any hosts that have been offline for a while and missed some updates, while avoiding re-doing tasks on every machine.

After the pre-script has been run, the update script next resynchronizes selected local configuration files with master copies on a central server. This is another place where a local administrator can step in and control the update process, by selectively excluding some configuration files. The resynchronization is again done with rsync, which this time uses a source of the form
This is synchronized with the destination "/", but this time the synchronization isn't exact:
  • Files present on the local machine, but not present in the master copy, are ignored.
  • Backup copies of any files changed during the resynchronization are saved into a date-stamped directory under /etc/oldconfigs
  • The rsync process looks for include/exclude lists in the local files /etc/CACHE_CONFIG.include and /etc/CACHE_CONFIG.exclude
The config file tree on the server might look something like this:
|-- Library
| |-- LaunchDaemons
| | `-- org.zebedee.zebedee.plist
| `-- Preferences
| |-- SystemConfiguration
| | `-- com.apple.PowerManagement.plist
| `-- com.apple.sharing.firewall.plist
`-- private
`-- etc
|-- bashrc
|-- csh.cshrc
|-- hostconfig
`-- profile

Next, the rest of the /common tree is synchronized with the master copy. The synchronization is exact in this case, with any extra local files being deleted. As before, immutable flags are cleared before the resynchronization and re-instated afterward. The local configuration variable CACHE_COMMON allows the local administrator to selectively skip this part of the update process.

After /common is resynchronized, the update script resynchronizes /local. This is a looser resynchronization, without the --delete flag, so that the local administrator can add extra things to the /local tree that aren't present in the master copy. The local administrator can also selectively exclude parts of the tree from synchronization by using the /etc/CACHE_LOCAL.exclude and CACHE_LOCAL.include files.

The update script next does an Apple softwareupdate, automatically installing all available appropriate updates. Once that's finished, it uses our pkgupdate script to update any locally-installed third-party packages. The list of tasks for pkgupdate is stored in /common/manager/update.pkglist .

Finally, the catchall post-update script (if any) is run and the update lock is cleared.

Friday, July 07, 2006

A Simple Package-Management Script

This is the third in a series of posts about a simple nightly update system for OS X.

In previous posts I've shown scripts to automate the installation of software under OS X, at least for software packaged in several of the common Mac formats (app bundles and pkg & mpkg bundles, possibly packed into a dmg image). In this post I'll talk about a simple package-management script that I'll use to keep packages up-to date, and to install new packages (and eventually, uninstall unwanted packages) in an automatic way every day.

The script checks a central repository (located on an anonymous ftp site) for a list of available packages, then looks at a master list of packages that should be installed on all standard computers. If any of the packages on the master list aren't installed locally, or if a newer version is present in the repository, those packages are fetched from the repository and installed. Updated packages can thus be distributed to all standard computers by simply adding updated packages to the repository, and new software can be added by taking the additional step of adding the new packages to the master list. Currently, I haven't implemented any mechanism for removing packages. This system is analogous to one we've been using successfully for many years on our Linux computers.

Note that this system is only intended to supplement the services already provided by Apple's own softwareupdate command. It's intended to install and maintain non-Apple software that won't be updated by softwareupdate.

I call the script "pkgupdate", and it looks like this:
Script pkgupdate:

use strict;

$PkgUpdate::noyaml = 0;
unless (eval "use YAML; 1") {
warn "YAML not found, falling back to simple config.\n";
$PkgUpdate::noyaml = 1;

use File::Temp qw/tempdir/;

my $configfile = "/etc/localconfig";
my @pkgtypes = ('app.tar.gz',

# Get list of actions:
my $pkglist = shift;
my $keep = shift;

# Load local configuration:
my $config;
my %simpleconfig;
unless ($PkgUpdate::noyaml) {
$config = LoadFile("$configfile.yml") or die "Cannot load config file $configfile.yml: $!\n";
} else {
%simpleconfig = simpleconfig($configfile);
$config = \%simpleconfig;

# Specify package URL:
my $pkgurl = "ftp://$config->{UPDATEMASTER}/$config->{UPDATEVERSION}/PKGS";

# Create temp directory:
my $cleanup = 1;
$keep && ($cleanup = 0);
my $tmpdir = tempdir( CLEANUP => $cleanup );
print "Using tmpdir $tmpdir.\n";

# Get list of available packages:
print "Fetching list of available packages... ";
`cd $tmpdir && curl -s -O $pkgurl/.pkglist`;
my %pkglist = ();
open (PKGLIST, "<$tmpdir/.pkglist") or die "Cannot open $tmpdir/.pkglist: $!\n";
while (<PKGLIST>) {
$pkglist{$2}{sum} = $1;
close (PKGLIST);
print "Done.\n";

# Create local package database, if it doesn't already exist:
unless ( -f "/etc/pkglist.sqlite3" ) {
print "Local package database /etc/pkglist.sqlite3 not found. Creating.\n";
print `sqlite3 /etc/pkglist.sqlite3 "create table package (sum text, file text)"`;
$? && die "Error creating /etc/pkglist.sqlite3\n";

# Get list of already-installed packages
my %pkglocal = ();
open (PKGLIST,"sqlite3 /etc/pkglist.sqlite3 \"select * from package\" |" ) or die "Cannot get list of locally-installed packages: $!\n";
while (<PKGLIST>) {
chomp( my ($sum,$file) = split(/\s*\|\s*/) );
$pkglocal{$file}{sum} = $sum;
close (PKGLIST);

# Process packages to install/update:
open (PKGLIST, "<$pkglist") or die "Cannot open $pkglist: $!\n";
PACKAGE: while (<PKGLIST>) {

# Skip comments and blanks:
/^(\#.*|\s*)$/ && next PACKAGE;

my ($action,$package) = split(/\s+/);
# Convert from glob syntax to regexp:
$package =~ s/\*/\.\*/g;
$package =~ s/\?/\./g;
# Find any matching packages:
my @matches = grep (/$package/, reverse sort keys %pkglist);
unless ( @matches ) {
warn "No matching package found for $package. Skipping.\n";
my $match = $matches[0];
# If multiple matches are found, choose one of them:
if ( @matches > 1 ) {
TYPE: for my $t (@pkgtypes) {
for my $m (@matches) {
if ( $m =~ /.*$t$/ ) {
$match = $m;
last TYPE;

# Find package type:
my $type = $match;
$type =~ s/.tar.gz$//;
$type =~ /^.*\.([^\.]+)$/;
$type = $1;

# Check to see if we already have this package:
unless ( $pkglocal{$match}{sum} eq $pkglist{$match}{sum} ) {

print "Executing action $action for package $match of type $type...\n";

for ($action) {
/install/ && do {
# Escape any spaces:
my $escmatch = $match;
$escmatch =~ s/(\s)/\\$1/g;
# Invoke the appropriate installer for this type:
`cd $tmpdir && curl -s -O $pkgurl/$escmatch`;
chomp( my $result = `md5 -r $tmpdir/$escmatch` );
$result =~ /^(\w{32}).*$/;
my $checksum = $1;
unless ( $checksum == $pkglist{$match}{sum} ) {
print STDERR "Error: Checksum mismatch for package $match. Skipping.\n";
if ( $escmatch =~ /^(.*)\.tar\.gz$/ ) {
`cd $tmpdir && tar xzf $escmatch && rm $escmatch`;
$escmatch = $1;
unless ( print `cd $tmpdir && /common/manager/$type-install $escmatch`) {
print STDERR "Error: $!, $?\n";
if ( $? ) {
print STDERR "Error installing package $match. Skipping local package list update.\n";
print "Updating local package database.\n";
print `sqlite3 /etc/pkglist.sqlite3 "insert into package (file,sum) values ('$match','$pkglist{$match}{sum}')"`;
if ( $? ) {
print STDERR "Error updating local package database.\n";
close (PKGLIST);

sub simpleconfig {
my $file = shift;

my %config = ();
open (FILE,"<$file") or die "Cannot open simpleconfig file $file: $!\n";
while (<FILE>) {
/^\#/ && next;
$config{$1} = $2;
close (FILE);
return %config;
The script draws local configuration information from either a YAML file (if this is available) or a simple file with the format "variable: value" if not. Only two parameters are used from the local configuration: UPDATEMASTER and UPDATEVERSION. From these, an ftp url of the form
is constructed. This is the location of the package repository.

The repository can contain software packaged as either dmg files or compressed tar archives of application or package bundles. The repository must also contain the file ".pkglist", which is a list of available files and their checksums. This list can be maintained on the server with a command like the following:
(find . -type f -print0 | sed -e 's/\.\///g'| xargs --null md5sum) > .pkglist
which can either be run by hand after dropping a new package into the repository, or run periodically through a cron job. Note that the "-print0" flag on "find" and the "--null" flag on xargs are necessary because many of the file names will contain spaces. I also pipe the "find" output through a sed command to strip off the leading "./" that would otherwise appear in the file names.

The resulting .pkglist file looks like this:
bd2ab919477b545bdc51209ae9ade105  Firefox
0c1920da27ead93b41958afa1c80f2fd Fugu1.2.0.dmg
504d1e037a639753e307e3d48e3f1f01 II2.dmg
72091614b4656fbdb0d5bc2f74abfe5a OracleCalendar.dmg
350a369b3ec955537822b4387b17b101 RDC103EN.dmg
5c38e7bbb389d129430c3af169745314 Thunderbird 1.5.dmg
f771d754c01023856444628514865e2e mozilla-mac-1.7.12.dmg
49369f4e72fe334c23d52489fc61110f OSXvnc1.71.dmg
The checksums will be used on the local computer to determine whether a given package/version has already been installed. The pkgupdate script maintains a local sqlite database of the packages it installs. After pkgupdate fetches the .pkglist file, compares the checksums of the available packages with the checksums of the already-installed packages and uses this information when deciding whether a package needs to be fetched from the repository and installed. I've used checksums for this to avoid having to try to figure out what version of a piece of software is packed into a given file. Filenames are unreliable for this purpose, and it would be complicated to unpack each package and try to find out what version number (or numbers, in the case of multi-packages!) was associated with it. Instead, I just assume that if two packages have the same checksum, they are the same version.

The pkgupdate script takes one argument: the name of the file containing the master list of packages to be installed on all computers. That file looks like this:
install OSXvnc*
install Firefox*
install Thunderbird*
install mozilla-mac*
install OracleCalendar*
install Fugu*
install II2*
install magicolor2430DL
install RDC103EN
with each line consisting of an action followed by a (possibly wild-carded) package name. Currently, the only available action is "install", although I'd like to add "remove" eventually, since we support this under the analogous system used on our Linux computers. The package names should match names in the .pkglist file.

When pkgupdate determines that it needs to install a package, it fetches the appropriate file from the ftp repository using curl, then invokes an external installer script (dmg-install, app-install or pkg-install), chosen based on the name of the downloaded file. (Compressed tar archives are first unpacked.)

If the installation succeeds, pkgupdate updates the sqlite database containing information about currently-installed packages. This lives in /etc/pkglist.sqlite3 . Currently, the database only contains filenames and corresponding checksums.

In the finaly posting in this series, I'll show how pkgupdate can be used as a component of a general-purpose nightly update system for OS X.

Automating pkg and app Bundle Installs

This is the second in a series of posts about a simple nightly update system for OS X

In the previous post I showed a script that mounted a dmg image, looked for packages and application bundles inside it, and called other scripts to install these packages and application bundles. In this post, I'll show the scripts I call "pkg-install" and "app-install", which do do actual work of installing software.

First, app-install, which installs application bundles by simply copying them into the /Applications directory.

Script app-install

use strict;
use File::Basename;

my $a = shift;

# Check to see if app of this version is already installed, and
# install if not.
my $appdir = basename($a);
if ( -d "/Applications/$appdir" ) {
chomp ( my $vnew =
`cat $a/Contents/Info.plist|grep -A1 CFBundleShortVersionString|tail -1` );
$vnew =~ s/\<\/*string\>//g;
$vnew =~ s/(^\s+|\s+$)//g;
chomp ( my $vold =
`cat /Applications/$appdir/Contents/Info.plist|grep -A1 CFBundleShortVersionString|tail -1` );
$vold =~ s/\<\/*string\>//g;
$vold =~ s/(^\s+|\s+$)//g;
if ( $vold == $vnew ) {
print "Application bundle of the same version ($vnew)";
print "is already installed.\n";
print "Skipping installation.\n";
print "Installing $a in /Applications...\n";
print `cp -r -p $a /Applications/`;
$? && die "Copy failed for application bundle $a\n";

The script takes a single argument, the name of the app directory, and starts by checking to see if an application of the same version is already installed. It does this by first looking for an app directory of the same name under /Applications and then, if it finds one, looking at the "CFBundleShortVersionString" key in the Info.plist files of the two application bundles. If the key is the same in both cases, the script assumes that the applications are identical and skips the installation. Unfortunately, there's no reliable way to tell if an already-installed version is newer than the candidate for installation, since version "numbers" are unique to each application and often contain more than just digits. Could a program reliably decide whether version 1.3-beta2 is newer than 1.3-development-1.7? The app-install script doesn't address this, and just assumes that if it's told to install a different version (older or newer), that's that the person invoking app-install really wants to do.

For pkg bundles (and multi-packages) I have a second script, called "pkg-install".
Script pkg-install

use strict;
use File::Basename;

my $p = shift;

# Check to see if pkg of this version is already installed, and
# install if not.
my $pkgdir = basename($p);
if ( -d "/Library/Receipts/$pkgdir" ) {
chomp ( my $vnew =
`cat $p/Contents/Info.plist|grep -A1 CFBundleShortVersionString|tail -1` );
$vnew =~ s/\<\/*string\>//g;
$vnew =~ s/(^\s+|\s+$)//g;
chomp ( my $vold =
`cat /Library/Receipts/$pkgdir/Contents/Info.plist|grep -A1 CFBundleShortVersionString|tail -1` );
$vold =~ s/\<\/*string\>//g;
$vold =~ s/(^\s+|\s+$)//g;
if ( $vold == $vnew ) {
print "Package of the same version ($vnew) is already installed.\n";
print "Skipping installation.\n";
print "Installing $p...\n";
print `installer -pkg $p -target /`;
$? && die "installer failed for package $p\n";

Again, the script first checks for an already-installed package of the same version. In this case, the script looks in the /Library/Receipts directory, where installer places metadata about packages after it installs them. If a package of the same version isn't already installed, the script then invokes installer to install the candidate package.

Automating Installs from DMG Images

This post is the first in a series documenting the development of a simple nightly update system for OS X. The system is intended to be a quick-and-dirty 80% solution for keeping standard OS X computers in our department up to date. We've been doing something similar for Linux for many years, and the OS X solution is an outgrowth of this.

I'll begin with the following script for installing software from dmg images, which I'll describe below:

Script dmg-install:

use strict;
use File::Basename;

my $DMG = shift;
$DMG =~ s/(\s)/\\$1/g;

# Attach and mount image:
print "Mounting image $DMG... ";
my $result = `yes| hdiutil attach $DMG | tail -1`;

# Get mounted volume information:
$result =~ /^.*\/dev\/disk.*(\/Volumes\/.*)$/;
my $mountpoint = $1;
$mountpoint =~ s/(\s)/\\$1/g;
print "$DMG is mounted on $mountpoint.\n";

# Find packages and application bundles:
my @apps = glob("$mountpoint/*.app");
my @pkgs = glob("$mountpoint/*.pkg");
push (@pkgs,glob("$mountpoint/*.mpkg"));

unless ( @apps or @pkgs ) {
my $list = `ls -al $mountpoint/`;
die "No application bundles or packages found in disk image.\nHere's what I see:\n$list";

for my $a (@apps) {
$a =~ s/(\s)/\\$1/g;
print "Installing application bundle $a...\n";
print `/common/manager/app-install "$a"`;
$? && die "Failed to install application bundle $a\n";

for my $p (@pkgs) {
$p =~ s/(\s)/\\$1/g;
print "Installing package $p...\n";
print `/common/manager/pkg-install "$p"`;
$? && die "Failed to install package $p\n";

print "Unmounting image $DMG from mountpoint $mountpoint...\n";
print `hdiutil detach $mountpoint`;

The dmg-install script takes the name of a dmg file as its only argument. The script uses hdiutil to mount the image, answering "y" to any questions asked during the process. Hdiutil returns information about the location of the mount point, and the script uses this to look there for any application bundles, pkg bundles or multi-packages. The list of application bundles is fed to a separate installation script called "app-install" (which I'll describe in a later post) and the list of pkg and mpkg bundles is fed to a script called "pkg-install" (also to be described later). After all bundles are installed, dmg-install unmounts the dmg image and exits.

Thursday, July 06, 2006

More on Disk Images, Resource Forks, Licenses

In a previous post I talked about disk images (dmg files) as a common mechanism for distributing software for OS X. I mentioned that dmg files can contain license agreements that must be accepted before the disk image can be mounted.

To understand the mechanism used for this, I'll need to talk about resource and data forks. In earlier versions of Apple's operating systems, each file actually had two parts, called a resource fork and a data fork. The file's data and metadata are split (in a sometimes complicated way) between the two forks. Often, the data fork is similar to what we'd normally think of as the file itself, and the resource fork carries metadata, in a binary format, about the file: an icon, a description, the name of an associated application, etc.. The resource fork is normally invisible to the user, and the filesystem takes care of keeping data and resource forks "connected". (When the file is moved to a new directory, for example, both forks are simultaneously moved.)

Resource forks are rarely used under OS X, but they're still supported by its HFS+ filesystem. You can get to a file's resource fork by appending "/rsrc" onto the file name. For example:
# ls -al /etc/bashrc
-rw-r--r-- 1 root wheel 250 Jun 15 10:48 /etc/bashrc
# ls -al /etc/bashrc/rsrc
-rw-r--r-- 1 root wheel 0 Jun 15 10:48 /etc/bashrc/rsrc
Weird, eh?

One problem with resource forks is that they can't easily be transported across the network (at least not to non-Apple servers). For example, If you ftp a file from an OS X computer to a Linux computer, only the data fork will be transported. Similarly, if you download a file from a web server, you'll only get the data fork, even if the file originally had a resource fork.

One way around this is to "flatten" the file by packing both resource and data forks into a new data file in such a way that they can be unpacked again later. This is often done with dmg files. A flattened dmg file can be unflattened with htiutil ("hdiutil unflatten file.dmg").

Licenses are often added to the resource forks of dmg files before they are flattened. Each component of a resource fork has a type (such as 'TEXT') and a numerical identifier. Before mounting a disk image, hdiutil looks in the dmg file's resource fork for a TEXT section with identifier 5002. This section holds the license agreement associated with the dmg. The perl script BuildDMG can be used to create dmg files, and it is capable of attaching a license as part of the process. BuildDMB hdiutil to unflatten the dmg file, then it uses the Rez command from the Xcode tools to compile the resource fork from a source-code description (here's an example). The dmg file is then re-flattened and ready for distribution.

To automate the mounting of a license-containing dmg file, you can either do something like this:
yes | hdiutil attach file.dmg
which will just answer "y" to any questions asked by hdiutil (e.g., "Do you accept this license?"), or you can unflatten the dmg file, clear its resource fork and reflatten it before trying to mount the image. One way to do this would be:
touch junk.rsrc
hdiutil unflatten file.dmg
/Developer/Tools/Rez junk.rsrc -o file.dmg/rsrc
hdiutil flatten file.dmg

chflags, Immutable files, Securelevel, Single-User Mode

It's often useful to mark a file as "immutable", requiring even the superuser to take some additional action before the file can be modified. Under Linux's ext2/3 filesystem this is implemented through the "attributes" commands (chattr, lsattr), using the immutable attribute. Under Windows, the "attrib" command provides similar functionality by allowing you to mark files as read-only.

Under Mac OS X, files can be marked immutable by using the "chflags" command. Chflags provides a way to attach several extended attributes (or "flags") to files and directories. Available flags are (from the man page):
arch    set the archived flag (root only)
opaque set the opaque flag (owner or root only)
nodump set the nodump flag (owner or root only)
sappnd set the system append-only flag (root only)
schg set the system immutable flag (root only)
sunlnk set the system undeletable flag (root only)
uappnd set the user append-only flag (owner or root only)
uchg set the user immutable flag (owner or root only)
uunlnk set the user undeletable flag (owner or root only)
There are also several aliases, such as "archived" for "arch". Each of the flags can be unset by prepending "no" (e.g., "nouchg" is the opposite of "uchg"). Flags set by chflags can be viewed with "ls -lo".

Chflags actually provides two slightly different "immutable" flags: uchg and schg. There are two differences between these flags. First, only the superuser can set the schg flag, but the uchg flag can also be set by a file's owner. Second (and perhaps more important), even the superuser may be unable to un-set the schg flag, depending on the current securelevel of the system.

Mac OS X inherits the concept of securelevel from BSD. The idea is that the kernel contains a variable called "securelevel". At boot time, securelevel has a value of 0, but this can be changed later. Once the securelevel has been raised, it cannot be lowered (even by the superuser) without a reboot. When Mac OS X goes into multi-user mode, it sets the value of securelevel to 1. You can see this with the "sysctl" command:
sysctl kern.securelevel
Going back to the immutable flags, the schg flag differs from the uchg flag in that schg cannot be unset unless the operating system is in securelevel 0. This means that if you set the schg flag on a file, you won't be able to unset it without booting into single-user mode (in which securelevel is still 0). You can boot into single-user mode by holding down the "command" (funky apple octothorpe/clover symbol) and "s" keys while booting.

Thursday, June 29, 2006

Software, Packages, Installer, and Softwareupdate

OS X provides several command-line tools for managing software packages. Note that software can be distributed for OS X in many different ways, and that packages are only one option. I'll start with some of the other alternatives.

Single Binaries

If you only need to install a single executable file that depends on no other files, you can just drop it into some directory on your search path. The default search path is "/bin:/sbin:/usr/bin:/usr/sbin".

Application Bundles

For software that requires several files, Apple offers a nice method of "drag and drop" installation. Under OS X, applications that are intended for use under the graphical interface are installed in the /Applications directory. Under this directory, each application has its own subdirectory, called an "application bundle", named with a ".app" extension (e.g., MyApp.app). Viewed from the finder, each of these app directories appears as a single file, although the directories contents may still be viewed by clicking the "gear" icon on the finder toolbar and selecting "Show Package Contents". These app directories can simply be dragged from another location (a CD, for example) and dropped into the /Applications folder. That's all that's needed to install the application. For information on the structure of an app directory, see The Structure of a Modern Bundle. Here's a partial listing of the contents of the Fugu application bundle (I've omitted a large number of language-specific files):
`-- Contents
|-- Info.plist
|-- MacOS
| `-- Fugu
|-- PkgInfo
`-- Resources
|-- ODBEditors.plist
|-- bookmark.png
|-- disconnect.png
|-- download.png
|-- downtriangle.png
|-- edit.png
|-- favorites.png
|-- favoritesprefs.png
|-- files.png
|-- fugu.icns
|-- generalprefs.png
|-- goto.png
|-- history.png
|-- home.png
|-- info.png
|-- knownhosts.png
|-- newfolder.png
|-- preview.png
|-- reload.png
|-- remotehistory.png
|-- remotehome.png
|-- remotetrash.png
|-- righttriangle.png
|-- symlink.png
|-- transfers.png
|-- trash.png
|-- uparrow.png
|-- upload.png
`-- zeroconf.png
At the top level is a "Contents" directory containing a plist file describing the application (specifying which icon to display, for example). Under this is a "MacOS" directory, containing the application executable itself ("Fugu") and a "Resources" directory containing files used by the application (such as icons, in this case).

Installer Packages

If an application requires interaction with the user during installation (to ask for default settings, for example), if tasks need to be done to prepare the system before installing the application, or if the software creator just wants to make the installation process a little prettier, the application may be delivered in the form of a package. Apple's native package format consists of a directory, named with a ".pkg" extension, and containing all of the files necessary to install the application. Like app directories, pkg directories are shown by the finder as a single file. For more information on the structure of a pkg directory, see The Anatomy of a Package.

Packages can be installed from the command line using the installer command. (Double-clicking on a pkg directory in the graphical interface will automatically launch installer.) More information about installer here.

Disk Images

Since it's usually inconvenient to copy a whole directory tree over the network, the most common way of distributing software is in the form of a single disk image (dmg) file. Each such dmg file contains an HFS filesystem containing a pkg or app directory tree. Dmg files can even by "internet-enabled" (by setting a flag in the file, using the hdiutil command), causing Safari to automatically mount, unpack and delete them after they're downloaded from a web server. Dmg files can also contain license agreements that will be displayed (and must be agreed to) before the image can be mounted.

Disk images can be mounted using the "hdiutil attach" command, like this:
hdiutil attach Firefox-
In the case of Firefox, the dmg contains a license agreement, which hdiutil displays. The user is prompted to accept the license, and then hdiutil proceeds to create a device associated with the disk image (e.g., /dev/disk1) and mount it under "/Volumes" (in this case "/Volumes/Firefox").


Finally, Apple provides a tool for updating software, called "softwareupdate". Softwareupdate is a command-line tool that can be used to look for available Apple software updates and/or install them. The command:
softwareupdate --list
will produce a list of available updates. Here's an example:
Software Update found the following new or updated software:
* QuickTime-7.1.2
QuickTime (7.1.2), 52650K [recommended] [restart]
* iTunesX-6.0.5
iTunes (6.0.5), 19340K [recommended]
The command:
softwareupdate --install --all
will find all available updates and install them. If one or more updates require a reboot, you'll be asked to restart the computer after the updates are installed. With the command
softwareupdate --schedule on
you can cause softwareupdate to automatically check for updated software. The check is done once per week, and is implemented by changing the current user's ~/Library/Preferences/com.apple.scheduler.plist file. This seems to imply that the checks will only occur when this user is logged in at the graphical console, although I haven't been able to find documentation verifying this.

For more information about software distribution, see this article.

Wednesday, June 28, 2006

Systemsetup and Networksetup

A couple of useful command-line utilities come bundled with Apple Remote Desktop (ARD). They can be used whether you're using ARD or not, and give you a convenient way to change some of OS X's defaults. The utilities are "systemsetup" and "networksetup", but you probably won't find either of them in your search path. They live under /System/Library/CoreServices in the directory RemoteManagement/ARDAgent.app/Contents/Support, where you'll find them with names like "systemsetup-panther" or "systemsetup-tiger". If you've ever enabled ARD (even if you've turned it off again later), you'll find that this directory also contains two symbolic links named "systemsetup" and "networksetup", which point to an apropriate "cat-named" version of each program.

With systemsetup (see the man page here), you can control or view a lot of the same settings that are available in the graphical "System Preferences" tool. For example, the command:
systemsetup -setsleep off

will disable power management for disks, display and CPU. The separate parameters "-setcomputersleep", "-setdisplaysleep" and "-setharddisksleep" can be used to turn power management on/off for each. Specifying a number of minutes instead of "off" will turn power management on. (These settings can also be changed by editing /Library/Preferences/SystemConfiguration/com.apple.PowerManagement.plist, but systemsetup provides a convenient interface that will -- presumably -- be independent of future structural changes in the plist file, and so can safely be used when developing scripts.)

Other examples, from the man page, are:

systemsetup -setdate 04:15:02
systemsetup -settime 16:20:00
systemsetup -settimezone US/Pacific
systemsetup -setnetworktimeserver time.apple.com

With networksetup (see the man page), you can configure or view network settings. For example, the command:

networksetup -getcomputername

will do the obvious. The command "networksetup -listallhardwareports" will show something like this:

Hardware Port: Bluetooth
Device: Bluetooth-Modem
Ethernet Address: N/A

Hardware Port: Built-in Ethernet
Device: en0
Ethernet Address: 00:16:cb:00:00:00

Hardware Port: Built-in FireWire
Device: fw0
Ethernet Address: 00:16:cb:ff:00:00:00:00

Hardware Port: AirPort
Device: en1
Ethernet Address: 00:16:cb:00:00:00

and "networksetup -getmacaddress en0" will return the wired network interface's MAC address.

Obviously, these commands only give access to a subset of the system's configuration parameters, but they're handy for many tasks.

Monday, June 26, 2006

Legacy StartupItems (OSXvnc)

Even though the current version of OS X deprecates StartupItems in favor of launchd, StartupItems are still supported, and some applications still use this mechanism for starting services. OSXvnc is one example.

StartupItems are analogous to the files found under init.d (or rc*.d) in operating systems that use System-V-style service-startup scripts. As with SysV init scripts, the application developer is expected to provide a script which will accept a few conventional command line arguments such as "start", "stop" and maybe "restart". The operating system then invokes the script with the appropriate argument to manage the service in question.

StartupItems provide an additional ability to control the order in which services are started. This is accomplished by listing tokens that are either "required" or "provided" by a given service. If a particular token is required to start a given service, the operating system will first start whatever other service provides that token.

As you'd expect, this meta-information about the service is provided in a Property List (plist) file. Each StartupItems service has an associated StartupParameters.plist file. In the case of OSXvnc, the file looks like this:

Description = "VNC Server";
Provides = ("VNC");
Requires = ("Resolver");
Uses = ("Core Graphics");
OrderPreference = "None";
Messages = {
restart = "Restarting OSXvnc server";
start = "Starting OSXvnc server";
stop = "Stopping OSXvnc server";

Note that this is an old-style, non-XML plist file. I'm not sure if this is a requirement for StartupItems, but all of the examples I've seen use this format.

The file says that the OSXvnc service will provide the token "VNC" (for potential use by other services that may depend on OSXvnc being running before they are started) and that it requires the token "Resolver" (meaning that the OSXvnc service should not be started until after the resolver service has started). The "Uses" property is similar to "Requires", but it specifies optional services. The operating system will try to start the list of services in the "Uses" property, but OSXvnc will still start even if these other services fail. Finally, an "OrderPreference" property can be used to fine-tune the startup order of services. Allowed values are First, Early, None, Late, or Last.

The operating system looks for StartupItem services in /System/Library/StartupItems and /Library/StartupItems. The former is for use by services provided by Apple, and the latter is for use by the local administrator. The startup order of services isn't affected by which directory the startup item is located in.

In addition to the plist file, you'll also need the "init" script mentioned above. The plist file and the init script for each service reside in a subdirectory of /System/Library/StartupItems or /Library/StartupItems. In the case of OSXvnc, this directory is /Library/StartupItems/OSXvnc. In the subdirectory should be an appropriate StartupParameters.plist file and an executable file (the init script) with the same name as the directory. Any other programs or data files needed by the service may also be located here.

At startup time, the operating system crawls through the subdirectories of /System/Library/StartupItems and /Library/StartupItems, looks at the "requires" and "provides" for each service, then builds up an ordered list of services to start. When it comes to OSXvnc, it will execute the command "/Library/StartupItems/OSXvnc/OSXvnc start" to start the service.
Update: Note that all files and directories from /Library/StartupItems on down must NOT be group- or world-writeable. Otherwise SystemStarter will refuse to run them.

StartupItems can be started/stopped at any time using the "SystemStarter" command. Some usage examples:

SystemStarter start VNC
SystemStarter stop VNC

Note that SystemStarter commands refer to the "Provides" tokens from the service's StartupParameters.plist file, rather than the name of a service's StartupItems directory (i.e., "VNC" versus "OSXvnc", in this case).

Also note that SystemStarter is run at boot time from /etc/rc, which is in turn run from launchd (like everything else).

For more information about StartupItems, see this Mac Developer article and this useful article at MacDevCenter.