So my webhost managed to mess up all htacess files and thus lock me out of my blog for a couple of weeks. I considered briefly posting via MySQL, but then dragged myself back into the realm of the sane and just waited it out. Apologies to the comments that were awaiting moderation…. My resolution is to actually blog more this year, so I’ve started trying out MarsEdit to see if being able to work offline improves the situation….

Another Macworld is over. To be honest, this one was a bit too hectic to really enjoy as a pure punter, and besides, I missed what was a brilliant match where India broke Australia’s record equalling winning streak in Perth.

Schoun Regan and I did the two day PowerTools conference on Advanced OS X Server, then Jeff McCune and I did a shorter presentation and demo on Puppet that seems to have provoked some interest in the MacEnterprise crowd.

The few other talks I managed to make it to were quite good, even given the difficulty of catering to the very wide range of technical ability in a Macworld crowd… I thoroughly enjoyed Andrina Kelly’s Lucid System Administration presentation, some of our Google techs were inspired by Kevin White’s Neutered Admins talk, and I heard lots of good reports about Greg Neagle and Philip Rhinehart’s System Imaging and Deployment two day session.

The two highlights of the week for me were firstly that Joel managed to work the optical inch into his Directory Services talk. Nice work….

Secondly would have to have been seeing Devo in the flesh. Thanks to Dave Pugh (you need to get that blog going Dave…), I now have much better photos than my mediocre camera gave….

Devo were simply awesome. I was prepared to have all my illusions shattered… but they put on an awesome performance….

Full props to Josh for getting this out there….

Go and read all about it now…

Seriously though, this is how imaging has to be done these days for Macs. Given that we have all these excellent tools now, and given that Apple keep messing us around by releasing hardware that isn’t quite compatible with the latest OS X point update and requires a specific kext or whatever… it’s just too much pain to manage monolithic images by hand anymore.

Seeing as how half the visitors to this site still seem to coming here via AFP-related searches in Google… this is something I’d noticed a while ago, but simply started adding more RAM as I had more lying around.

Find operations on AFP servers can really slow down the system. Apple have now produced a nice techinfo article with a really simple fix.:

Mac OS X Server (Universal) 10.4.7 and later: May become unresponsive when clients search mounted network volumes

Short, and to the point, but I’m still not sure if there’s a decent reason why they’re not telling us to put this sort of a change in /etc/sysctl.conf instead…

So after a couple of people requested this on the MacEnterprise mailing list….

This is a really simple AppleScript Studio application that allows you to create a Mobile User Account with a local home directory other than “/Users/nigelkersten”.

To get started….

• Open up the Xcode project.
• Edit “ldapDomain” and “pathToHomes” to reflect your LDAP domain and where you want the home directories to go to.
• Make sure you leave the trailing slash in for ‘pathToHomes’, otherwise you’ll end up with home directories like: “/Volumes/Storage/Usersnigelkersten”
• Compile it and run it.

When you launch it, it will spend a few seconds collecting usernames so that you can auto-complete the combo box.
I’ve left a grep statement in there to illustrate how you can filter out unwanted usernames:

do shell script "dscl  " & ldapDomain & "  -list /Users | egrep -v '^(z|Z)[0-9]{7}' |egrep -v '^vpn_*' | grep -v admin  > /tmp/ldapusers.txt"

the section:
egrep -v '^(z|Z)[0-9]{7}' filters out our student ids, which are ‘z’ followed by 7 numbers.
If you wanted to pull that filter out, just change it to:

do shell script "dscl  " & ldapDomain & "  -list /Users | egrep -v '^vpn_*' | grep -v admin  > /tmp/ldapusers.txt"

and the same goes for the other filters, which remove admin accounts in my area and the vpn user accounts that get created.

No guarantees about this app. It’s a rather quick and dirty hack, and you’re welcome to do whatever you want with it. I’d appreciate seeing any improvements you make to it however.

You can download it here

So something I find that I need to do quite often is to log in as a specific user, and I don’t always want to reset their password.

While AFP has a really useful function where you can allow admins of an AFP server to masquerade as a specific user by using the admin password, no other services really offer that same kind of functionality.

What I tend to do in these situations is to swap the Authentication Authority around. I keep a test user account around that I know the password to, and I swap its Authentication Authority with the user I need to login as.

As the Authentication Authority describes which Password Server slot contains the password for a given user, by swapping them around you are effectively swapping the passwords. If you’re in a nice Kerberized environment (and if you’re not, you should be…) then you’ll notice that this doesn’t actually swap the Kerberos passwords around. This is because in a standard Open Directory setup, there are actually two password stores, one for Password Server and one for Kerberos. Apple have done some nifty stuff behind the scenes to keep the two in sync when users change their passwords, but by swapping Authentication Authorities around, you’re not modifying the Kerberos passwords at all.

Anyway, here’s the script I use to swap authentication authorities around. I tend to save it as something like “swap_auth_authorities.sh”, and then you’d use it like:

./swap_auth_authorities.sh -n /LDAPv3/my.od.domain -a diradmin -p diradminpass  testone testtwo

which would swap the Authentication Authorities for the users “testone” and “testtwo”.

As always, don’t blame me if you blow up your server, and you should definitely try this on test accounts first…. You can use the command “/usr/libexec/chkpasswd testone” to test the passwords for each user. If you get “Sorry” back, the password is wrong, if you get nothing back, the password is correct.

Love that famed Apple user friendliness :)

Script follows, or you can download it here:

#!/bin/bash
#

directorynode="/LDAPv3/127.0.0.1"
adminusername="diradmin"

export PATH="/usr/bin"

show_usage() {
echo "This script will swap the authentication authorities of two users in a directory node."
echo usage: swap_auth_authorities [-h] [-n directorynode] [-a adminusername] [-p adminpassword] username1 username2 ...
echo
echo "-h help (this output)"
echo "-n directory node name (default: /LDAPv3/127.0.0.1)"
echo "-a directory admin username (default: diradmin)"
echo "-p directory admin password"
echo "username1 and username2 are the users you wish to swap authentication authorities for"
echo
echo "Note: If the admin password is not specified, you will be prompted for it a number of times."
}

show_password_prompt() {
  if [ ${#adminpassword} -eq 0 ]; then
    echo "Enter the password for $adminusername"
  fi
}

while getopts  "hn:a:p:" flag; do
  # echo "$flag" $OPTIND $OPTARG
  case "$flag" in
    h) show_usage; exit ;;
    n) directorynode="$OPTARG" ;;
    a) adminusername="$OPTARG" ;;
    p) adminpassword="$OPTARG" ;;
    ?) echo >&2 "Illegal option '$OPTARG'"; exit 1 ;;
  esac
done
shift $(($OPTIND -1 ))

if [ $# -ne 2 ]; then
  echo >&2 "Error: You should only pass two additional parameters, the usernames whose authentication authorities are to be swapped"
  echo
  show_usage
  exit 1
fi

if [ ${#adminpassword} -eq 0 ]; then
  authstring="-u $adminusername -p"
  echo "You will now be prompted several times for the password for $adminusername"
else
  authstring="-u $adminusername -P $adminpassword"
fi

show_password_prompt
authority1=$(dscl $authstring $directorynode -read /Users/$1 AuthenticationAuthority | sed 's|^AuthenticationAuthority: ||g')

# check to see if users exist or permission denied.
if [ $? -ne 0 ]; then
  case $? in
    71 ) echo "User $1 not found in $directorynode"; exit 1 ;;
  esac
fi

# dscl really should return an error code in these cases...
if [ $(echo $authority1 | grep -c "Data source ($directorynode) is not valid.") -gt 0 ]; then
  echo "There is a problem with either your admin username/password combination, or the specified directory node"
  exit 1;
fi

show_password_prompt
authority2=$(dscl $authstring $directorynode -read /Users/$2 AuthenticationAuthority | sed 's|^AuthenticationAuthority: ||g')

# check to see if users exist or permission denied.
if [ $? -ne 0 ]; then
  case $? in
    71 ) echo "User $2 not found in $directorynode"; exit 1 ;;
  esac
fi

show_password_prompt
dscl $authstring $directorynode -create /Users/$1 AuthenticationAuthority "$authority2"

show_password_prompt
dscl $authstring $directorynode -create /Users/$2 AuthenticationAuthority "$authority1"

echo "The authentication authorities for $1 and $2 have been swapped."
echo "Note that Kerberos authentication will continue to use the old passwords."
echo "If you re-run this command, the authentication authorities will be swapped back again."

“Tuning” is in quotes because… well you’ll see once you read the article. :)

Go check it out at afp548.com, which you should all be reading religiously anyway. :)

I’ve just added a link to it in the side column, but it’s well worthwhile checking out Greg Neagle’s Managing OS X blog.

There are getting the screensaver to work over the loginwindow, several, really, good posts on using Portable Home Directories without Open Directory.

AFP. It ain’t so bad.

From looking at the mailing lists and discussion boards, you’ll see that a reasonable number of people seem to be having problems with their AFP servers under OS X Server 10.4.x, particularly under heavy load with network home directories, and particularly in terms of stability. I used to be one of those people, but the solutions presented in this article have (touch wood) resolved the vast majority of my issues.

Whereas under 10.3 I used to get a symptom where the whole server would lock up and not even be pingable, the symptoms I’ve seen under 10.4.x have been entirely related to the AFP server. Login times will get longer and longer, access speed becomes slower and slower, and eventually the AFP server enters a death spiral, where the only recourse seems to be restarting the service.

I’m going to quickly cover some of the more commonly known fixes and workarounds that I’d tried before these major solutions.

1) Make sure DNS is working with proper forward and reverse entries for your servers at the bare minimum, and preferably all your client machines as well.

2) Redirect ~/Library/Caches to a local folder for network home directory users.
This makes quite a big difference. If you start watching your AFP server logs, you’ll notice that your network home directory users hit the cache a lot, particularly for apps like Safari. We do this with a login hook that redirects ~/Library/Caches to /Library/Caches/username/.

3) Disable creation of .DS_Store files for network mounts.
There is an Apple techinfo article up about this. From debugging, it looks like a corrupt .DS_Store file can create all sorts of problems, particularly if you have a setup like mine, where students have read-only access to certain network folders, and lecturers have write-access. I’ve used MCX to push out the above setting, and then run a command like this to delete all existing .DS_Store files on the sharepoint. The problem with this solution is that your users will get a bit narky about no longer being able to have window settings ‘stick’ across sessions.

4) Check the integrity of your filesystem.

5) Put more RAM in your server.
I was previously looking at swapfile creation and vm_stat to work out whether my AFP servers needed more RAM, however the problem seems to be that the AFP server tries to avoid swapping to disk, and so these aren’t good metrics to work from. As soon as I stuck a couple more gig into my servers, they started using it…

6) Move heavy users to Mobile Accounts with Portable Home Directories.
This isn’t applicable in all situations, and there’s no way I could move all my students to Mobile Accounts, but for my staff members who use Office and Mail heavily, performance on a network home directory isn’t that great, even on a fast connection. If you have users who primarily use a single machine, Mobile Accounts where you control the synchronisation settings via MCX give you the best of both worlds.

7) Increase the sysctl parameters for max files and max files per process.
This is a more general tuning tip for OS X Server, but as Rob Middleton pointed out to me, without it, if you have AFP error logging on, it helpfully logs several hundred lines per second to inform you that it can’t open files… filling up your primary partition rather quickly…
Create /etc/sysctl.conf with the following parameters:

kern.maxfiles=200000

kern.maxfilesperproc=50000


If you want to apply those settings immediately, rather than waiting for a reboot you can do:

for i in $(cat /etc/sysctl.conf); do sysctl -w $i; done


Now onto the more interesting fixes…

1) Tweaking the WAN threshold and packet size on the clients

If you run this command on a client machine, you can see various AFP client tuning parameters. Some of your settings may be slightly different to these.

  nigelkersten@zombie: ~ $ defaults read -g com.apple.AppleShareClientCore
  {
      "afp_active_timeout" = 0;
      "afp_authtype_show" = 0;
      "afp_cleartext_allow" = 1;
      "afp_cleartext_warn" = 1;
      "afp_debug_level" = 6;
      "afp_debug_syslog" = 0;
      "afp_default_name" = "";
      "afp_idle_timeout" = 0;
      "afp_keychain_add" = 1;
      "afp_keychain_search" = 1;
      "afp_login_displayGreeting" = 1;
      "afp_maxDirCache" = 60;
      "afp_maxFileCache" = 60;
      "afp_minDirCache" = 5;
      "afp_minFileCache" = 5;
      "afp_mount_defaultFlags" = 0;
      "afp_no_kQueues" = 0;
      "afp_no_volChange_caching" = 1;
      "afp_prefs_version" = 2;
      "afp_reconnect_allow" = 1;
      "afp_reconnect_interval" = 10;
      "afp_reconnect_retries" = 12;
      "afp_ssh_allow" = 0;
      "afp_ssh_force" = 0;
      "afp_ssh_require" = 0;
      "afp_ssh_warn" = 1;
      "afp_use_default_name" = 0;
      "afp_use_short_name" = 0;
      "afp_voldlog_skipIfOnly" = 0;
      "afp_wan_quantum" = 8192;
      "afp_wan_threshold" = 30;
  }
  

Of particular interest are these two settings, ‘afp_wan_quantum’ and ‘afp_wan_threshold’. The values I’ve shown here should be the defaults, but you may have a setting of 0 for both of them. These are used by the client to work out whether a particular AFP connection is over a LAN or WAN connection. If the latency of a given connection is higher than the value in afp_wan_threshold, then the data chunk size drops from 128KiB (the default for a LAN connection) to 8KiB, the setting shown here.

The problem seems to be that this default threshold setting is way too low, and once the AFP server starts experiencing moderate load, your LAN clients start using the WAN data chunk size. Although smaller chunks are desirable for slow connections, they induce an overhead on the server in terms of processing as the server is dealing with 16x the number of chunks, and reduce overall throughput. A symptom of this is high CPU usage.

The good news is that we can change these settings. The values you choose are up to you, and depend largely upon whether you actually do have WAN clients to support, so I’m going to suggest two scenarios:

First scenario: No WAN clients (cause AFP sucks over slow connections anyway :) )

    afp_wan_quantum = 131072
    afp_wan_threshold = 1000
    

This is a bit of a shotgun approach. Bring the latency threshold way up, and even if the clients reach this threshold, force their chunk size to be the same as that of a LAN client. This is what I’ve done in my environment, mainly because I wanted to make sure that I could rule out this issue. As time goes on, and once I work out the correct way to measure latency for a client (see the footnote at the end), I’ll come up with more sane values, but at this stage, I just needed to stop my lecturers from revolting en masse due to AFP stability issues.

Second scenario: Some WAN clients who use AFP (I feel their pain…)

  afp_wan_quantum = 8192
  afp_wan_threshold = 200
  

This is very much a guesstimate on my part. I’ve done a few tests at a threshold of 100, and I’ve found that LAN clients were still getting the WAN chunk size when I set the threshold at 100. Again, see the footnote at the end, as I’d like to find out a way of getting an accurate picture of the latency a client is experiencing.

So as far as this setting goes, the good news (for me) is that it has almost entirely resolved performance issues with the AFP server under moderate load. I’ve also been experimenting with the maxFileCache and maxDirCache settings, and even when I’ve been increasing them from 60 to 6000, I’ve been unable to replicate any stale cache issues. If you so desire, try playing around with those settings.

Applying these settings to a client machine.

Well, there are a couple of options you have here.

1) Apply the settings manually.

One way would be to use a LoginHook to set these parameters using the defaults command. ie, the first scenario above could be done with the following two commands:

    defaults write -g com.apple.AppleShareClientCore -dict-add afp_wan_quantum -int 131702
    defaults write -g com.apple.AppleShareClientCore -dict-add afp_wan_threshold -int 1000
  

The “-g” refers to the fact that we’re writing these settings into the Global Domain, (for a user, at ~/Library/Preferences/.GlobalPreferences.plist), -dict-add means you’re adding a key value to a dictionary (named “com.apple.AppleShareClientCore”), and -int means you’re adding an integer value.

This may be the best solution for your environment, and is a good place to start testing before you move onto the next method…

2) Use MCX to manage the settings.

This is the way I’ve done it, as it has some nice side effects, and is more The Apple Way ™. As with all such settings, you can choose to do this at the user or group level. I’ve chosen to do it to my two main groups that cover all my staff and all my students.

• Open up Workgroup Manager, and choose the group or user you wish to apply these settings to. Click on the toolbar item ‘Preferences’ and then the tab ‘Details’ to the right.
• Click on the ‘Add’ button, and navigate to your own home folder. Choose “.GlobalPreferences.plist”. You’ll now see it appear in the Preferences pane.
• Double click on the .GlobalPreferences item to edit it. You’ll notice that the settings have been put into ‘Often’ rather than ‘Always’ or ‘Once’. I’ve had some really odd behaviour with trying to get the global defaults domain to be managed ‘Always’, and ‘Often’ has been working happily for me, so let’s leave it set that way.
• If you have any other items other than the ‘com.apple.AppleShareClientCore’ dictionary, delete them, unless of course you wish to manage those settings for your users as well. Click on the triangle next to the AppleShareClientCore dictionary to expand it.
• Here you can see the relevant settings. Change the afp_wan_threshold and afp_wan_quantum values to the ones you’ve decided to use. Click on ‘Apply Now’.

Done! A really nice side effect of doing things this way is that you can now centrally manage a bunch of other afp connection settings, and perhaps most usefully, you can turn on AFP client side debugging using MCX for all your users. The AFP server itself isn’t particularly useful at giving debugging info, but the client is actually quite good. The footnote at the bottom will explain how to turn on client-side debugging.

Since I applied these changes, I’ve seen a drastic change in stability and performance under load. However… resolving this issue has pointed to another problem with Apple’s default settings, and I’ve noticed that the server is still slowing down under much heavier load, and still not using all the resources available to it.

There are a few oddities here though. It seems like global preferences shouldn’t take effect until the home directory is actually mounted, but perhaps by putting the prefs into MCX, we’re having them take effect before the actual home directory has mounted… ?

If anyone goes down the manual path setting, I’d like to hear if you’re finding these settings aren’t taking effect

2) Tweaking the maximum # of threads on on the server

The AFP server saw a change in 10.4, where permissions are now calculated by spawning a new thread for each connection with the effective uid of the connecting user. The problem is….

    root@server: ~ $ serveradmin settings afp:maxThreads
    afp:maxThreads = 40
  

That seems kind of low given the above, right? Apparently we should have at least one thread per client session, and this would include all your automounts…

I’ve set mine to 600 here like this:

    sudo serveradmin settings afp:maxThreads=600
  

This is something that again will depend upon what else your AFP server is doing (ideally not much…) and how many clients and automounts you have.

Footnote: Working out the latency of a client connection and AFP client side debugging.

You may have noticed two debug settings in the AppleShareClientCore dictionary as you were applying those settings.

    afp_debug_level = 6
    afp_debug_syslog = 0
  

If you set afp_debug_syslog to 1 (true), and add the following line to /etc/syslogd.conf,

  *.debug                                                 /var/log/debug.log
  

then you’ll see a wealth of debugging info (depending upon the value from 1 to 8 you’ve set for afp_debug_level) go to /var/log/debug.log

This isn’t something you should just willy-nilly turn on for all your clients, as tempting as it may be to finally have debug info for AFP… Rather use it to have a look at what goes on when a client connects to an AFP mount.

You’ll notice if you have the debug level up quite high that you’ll see a bunch of times reported to the debug log. I have yet to work out how to interpret those times to get the latency of a given client connection, and this is something I’d like to know so that I can tune ‘real’ values for my afp_wan_threshold.

So go forth! and work out how to do it. :)

As always, I’m not responsible for your server spontaneously combusting, your mileage may vary and this article may contain traces of nuts.

Many thanks must go to the afp548.com posse, Joel n Josh for extensively talking through these issues with me, as well as bringing them to my attention…

Recently I finally bit the bullet and migrated my 10.3.9 Open Directory Master to 10.4.3, and ran into a couple of problems along the way…

Although I’ve been hearing the odd report about a successful upgrade, I decided I’d follow something similar to Andrina’s article on afp548.com on how to dump LDAP, Password Server and Kerberos, and then re-import those to a clean 10.4 Open Directory Master.

1. The first problem I ran into was Andrina’s suggestion to use mkpassdb -mergeparent. Doing things this way will generate a new RSA key for the entire Password Server database, which is problematic for my setup. I have way too many mobile users (we’ve been using mobile accounts since they were introduced in 10.3) who have an original_authentication_authority property in their local mobile user record in NetInfo. If the RSA key were to change, I’m pretty sure these accounts would essentially be orphaned, and I just decided I couldn’t take that risk, so I instead used mkpassdb -mergedb, which will replace the entire Password Server db from the 10.3.9 backup, rather than merging it with the clean 10.4.3 database.
2. Another really really annoying problem I ran into regarded the use of slapcat and slapadd to dump and import the LDAP database. I have quite a few ‘email only’ user accounts that do not have a homedirectory property set. As I’ve created these via Workgrop Manager, these are of the type ‘posixAccount’, and the LDAP schema definition for posixAccount requires that all such accounts MUST have a home property. slapadd refused to import these users.

   I had at least three options once I realised this was an issue.

   1. Reboot my existing 10.3.9 server, and use Workgroup Manager to export all the user records, then import them to the 10.4.3 server, and then use the ldap tools to overwrite the authentication_authority records for those accounts. As they had already been created, the ldap tools should have had no problem overwriting properties.
   2. Use ldapmodify/ldapadd to add the accounts from the ldif dump, as they ignore the schema checking that slapadd does. I’d run into some issues with the PWS import, so I was having trouble authenticating to the LDAP directory via password, and I also ran into some issues with the Kerberos import, mainly to do with not having an ldap principal for the OD master so I couldn’t authenticate that way either.
   3. Modify the schema for posixAccount such that it no longer required a home attribute. This seemed to be the path of least resistance, so I went this way. To do this, edit /etc/openldap/schemas/nis.schema, and change the posixAccount definition:
      

      objectclass ( 1.3.6.1.1.1.2.0 NAME 'posixAccount' SUP top AUXILIARY
              DESC 'Abstraction of an account with POSIX attributes'
             MUST ( cn $ uid $ uidNumber $ gidNumber $homeDirectory )
              MAY ( userPassword $ loginShell $ gecos $ description ) )
      

      to:

      objectclass ( 1.3.6.1.1.1.2.0 NAME 'posixAccount' SUP top AUXILIARY
              DESC 'Abstraction of an account with POSIX attributes'
             MUST ( cn $ uid $ uidNumber $ gidNumber )
              MAY ( userPassword $ loginShell $ gecos $ description ) )
      

      Once I’d done this, slapadd happily added all my email-only users.

3. Kerberos. This caused me no end of pain. I initially didn’t want to simply do mkpassdb -kerberize, as I wanted to keep all my existing host and service principals for all my other servers. This was me being kind of lazy, but I also wanted to keep the option to simply roll back to my 10.3.9 OD Master if the migration didn’t work out properly.

   In the end, after struggling with a bunch of different things, particularly the new kinds of principals in 10.4, I just realised I was being stubborn, and created an entirely new Kerberos setup. Part of the reason I was being so stubborn was due to me forgetting just how nice mkpassdb -kerberize is in 10.4. It’s a couple of minutes at most to generate an entirely new Kerberos setup from PWS/LDAP, and then it’s simply a matter of running sso_util on the other servers, and restarting any services that use Kerberos. Definitely the path of least resistance…

So as part of my eternal quest to never ever have to deploy services that require their own authentication database, I’ve been poking around for a good Wiki for OS X that can authenticate to Open Directory without too much faffing around.

As my main aim for setting up a Wiki was to provide a space where we could have collaborative documentation, I was quite pleased to run across DokuWiki, and even more pleased when I realised that it had a built in ACL framework that supported LDAP authentication.

If you scroll down to the bottom of the auth_ldap documentation, you’ll see I’ve put in the settings needed to get authentication against Open Directory working.

Works like a charm, and the syntax is easy enough that it is making documentation nice and simple.

Props to Andreas Gohr for such a nice software package.