My first (product) baby – Puppet Enterprise 1.0

Today we (Puppet Labs) announced the release of Puppet Enterprise 1.0.

This has been the big project I’ve been working on since I left Google for the life of being a product manager at a startup software company. Lack of sleep, worrying about names, trying impose order on the birthing process…. it’s not that unlike real babies.

I’m proud of Puppet Enterprise.

Puppet is an awesomely powerful tool, but it does take some effort to get a scalable deployment going. It’s not a lot of work if you’re a reasonably experienced sysadmin, but some people simply don’t have the time.

We’ve launched with support for RHEL, CentOS, OEL, Debian and Ubuntu, and I’m particularly looking forward to our upcoming Mac OS X version.  I may have left the world behind, but all my Mac IT peeps still hold a special place in my heart.

There’s barely time to take a breath now 1.0 is out, as I’m heading off to FOSDEM in a couple of days to run the configuration management devroom with James Turnbull.

I’m planning to conduct some in-depth quality control inspections of Belgian beer while I’m in Brussels. Suggestions for inspection locations are more than welcome.

Farewell Google, Hello Puppet Labs

Last week I resigned from Google.

It’s been a crazy 4 years. Helping set up the largest Mac deployment I’ve ever dealt with, getting into Puppet, learning a lot of Python, discovering how robust you can make bash, moving to the Linux teams, learning from a lot of awfully smart people, and dealing with the fallout from the Operation Aurora attacks earlier this year… it’s hard to believe it’s only been 4 years in some ways.

It felt right to put something up here about this, but don’t expect any earth shattering revelations or horror stories.

I thoroughly enjoyed working at Google. I loved working with so many talented people, and I loved getting to work on infrastructure at scale that was developed by smart people.

However, the time was right to leave. The endless suburbia of the Bay Area was starting to get to us, and I’ve been very close to the Puppet Labs folks for a long time now, so when they offered me the position of Product Manager, and a chance to finally put my money where my mouth is about Puppet… well I couldn’t turn it down.

We have a lot of work to do at Puppet Labs, but it’s exciting work, as I honestly feel that we have a great chance of helping to fundamentally improve our industry. What more could I ask for out of a job?

Using crankd to run Puppet on network events

Gary Larizza has put together a great article that covers setting up crankd and Puppet to automatically apply system updates to Mac laptops when they appear on the corporate network.

This is exactly what Chris and I wrote it for initially, so it’s great to see this info out there.

Now I have somewhere to point Greg if he bugs me about it again. :)

pymacds – managing DirectoryServices with Python

[updated - 0.3 out with more useful wrapper methods]

I just added the pymacds module to the pymacadmin project.

To install the eggs for Python2.5 (OS X 10.5 only comes with 2.5) or Python2.6 (OS X 10.6 comes with both), run something like:

curl -O
sudo easy_install-2.5 pymacds-0.3-py2.5.egg


curl -O
sudo easy_install-2.6 pymacds-0.3-py2.6.egg

Once you’ve done this, you can use it like:

nigelk$ python2.5
Python 2.5.4 (r254:67916, Feb 11 2010, 00:50:55)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymacds
>>> pymacds.UserAttribute('nigelk', 'NFSHomeDirectory')
>>> pymacds.GetSearchNodes()
['/Local/Default', '/BSD/local', '/LDAPv3/my.ldap.node']
>>> pymacds.GetContactsNodes()

There are a few other useful methods, listing them here rather than going through them one by one. We’re particularly fond of the Ensure* methods for triggering DirectoryService node addition/removal on network events via crankd.

DSQuery(dstype, objectname, attribute=None):
DSSet(dstype, objectname, attribute=None, value=None):
DSDelete(dstype, objectname, attribute=None, value=None):
UserAttribute(username, attribute):
GroupAttribute(groupname, attribute):
AddUserToLocalGroup(username, group):
RemoveUserFromLocalGroup(username, group):

Code contributions welcome.

python subprocess over shell

One of my common gripes is when people struggle with complicated shell scripts that would be much simpler in a scripting language like Python, Ruby or Perl. I used to abuse PHP for this, but saw the light.

If you’re talking about replacing shell scripts, all of these are pretty much equivalent, but I don’t really do Perl for no particular reason, I’m a big fan of Python for general purpose work just because of the rich module system, and at least “dozens of engineers” use it at work..

The reason we generally write shell scripts is because we want to execute a bunch of external processes in an automated way.

In this area you can whip something basic up fastest in shell, yes, but at some point you’re going to have to repay that technical debt if you need to get past a certain point.

Besides, it’s not like shell scripting always stays simple and easy… and the overhead of moving to a more powerful language isn’t that huge.

If I’m sure the scope of a script will be small, or I don’t have the option of moving to structured data format for input and output with external commands, or I don’t have to futz around with arrays, I’ll stick with bash.

But, if I want to work with dictionary objects or talk in a protocol like LDAP or model things as objects, or need complex handling and passing around of stdin/stdout/exit statuses, or know some module handles lots of edge cases for me, I’ll move to Python or Ruby. I quite like both, but feel that Python is more utilitarian, and simply due to whitespace enforcement and extensive linters is a good fit for code that may need to be picked up and understood quickly by a co-worker.

When it comes to getting started with Python, I still suggest Dive into Python for people. Just flipping through Chapters 1-3 equips you with an awful lot.

Anyway, some people think Python is hard, but it’s not really. I think of Python as being extremely utilitarian, which makes it a great fit for sysadmin work.

Someone posted on the MacEnterprise list a question about working out what PPD was in use by each printer on the system in OS X, and someone gave a good shell example using the usual suspects of for, grep and awk.

There’s nothing really wrong with doing this, but I’ve come to distrust parsing non-structured output if I need to keep this solution working across many multiple versions of operating systems, or even between OS X major versions given Apple’s history. One of the big advantages of moving to Python is being able to parse and manipulate property lists, which can get really painful in shell. You end up writing out lots of temp files or giving up on dealing with error conditions.

Anyway, so lets see what it’s like executing a command in Python with the subprocess module. We’re asking System Profiler for printer info, and telling it to return the output in XML plist format.


import subprocess

command = [‘system_profiler’, ‘-xml’, ‘SPPrintersDataType’]
task = subprocess.Popen(command,

(stdout, stderr) = task.communicate()
print stdout

To skim through those lines, we’re

  • setting the python shebang
  • importing the subprocess module
  • defining a list called ‘command’ to store the command we want to run
  • creating a subprocess Popen object to run our command called ‘task’
  • setting standard out and standard error to go to our own pipes
  • getting standard out and standard error from the task.
  • printing standard out

This really isn’t much work, and if you really only wanted this functionality, there are other convenience functions that you can use to make this even shorter, or write your own convenience function.

But we have all sorts of options now.

We can send standard input to the task:

(stdout, stderr) = task.communicate(stdin)

We can ask what the exit status is easily:

status = task.returncode

and if we get None we know the process hasn’t terminated yet.

And if we want to do something a bit more complicated we can search through the structured data plist output easily like:


import plistlib
import subprocess

command = [‘system_profiler’, ‘-xml’, ‘SPPrintersDataType’]
task = subprocess.Popen(command,

(stdout, stderr) = task.communicate()
printers = plistlib.readPlistFromString(stdout)
printers = printers[0][‘_items’]

for printer in printers:
  print ‘Name: ‘ + printer[‘_name’]
  print ‘PPD: ‘ + printer[‘ppd’]

This is easier to extend than the standard for/grep/awk/sed equivalents tend to be, and much easier for a co-worker to pick up and understand. It comes close to documenting itself, just needs some comments about the structure of Apple’s output, and some try/except blocks in case the output is malformed or does change.

Seeking input for possible Mac IT Conference

The MacEnterprise steering committee has been talking about doing this
for way too long, but the recent lack of significant IT tracks at WWDC
has spurred us into action.

MacEnterprise is planning to partner with various other groups to get
a Mac IT focused conference started.

This is all very much up in the air, and at this stage we’re seeking
input as to how the community would like this conference to be

At this stage, we would like you to provide input on Google Moderator
as to ideas for the conference. You can submit ideas as well as vote
on other ideas here:

Additionally, there is some discussion going on on Twitter, under the
#MacITConference hashtag.

Once we get a little bit better idea of the structure, we’ll be
calling for speakers and looking for sponsorship partners.

Profiling puppetmasterd with ruby-prof

So I’m not hugely happy with the CPU consumption of puppetmasterd under heavy load, and so I’ve been trying to work out where the bottlenecks lie.

Unfortunately I’ve yet to find a smoking gun, but here’s a reasonably simple way to produce profiles of puppetmasterd.

  • Install ruby-prof from gems.
  • Stop any apache/mongrel/nginx instances of puppetmasterd you may have running
  • Edit /usr/sbin/puppetmasterd and replace the last few lines with:
    require ‘rubygems’
    require ‘ruby-prof’

    result = RubyProf.profile do
      require ‘puppet/application/puppetmasterd’

    printer =‘/tmp/ruby-profile.html’, ‘w’) do |file|
      printer.print(file, {:min_percent => 10, :print_file => true})

  • Start a webrick puppetmasterd with –no-daemonize
  • Do a client run against it
  • Hit Ctrl-C to interrupt your puppetmasterd
  • wait for the html output to be generated

It’s worth filtering the min_percent value. Without it, I ended up with 300+MB HTML files with no images that took my dev server a long time to write to disk. With it, I end up with a couple of megs.

You can see an example output at:

with the interesting thread being at:

Interpretation suggestions welcome :)


Brice had a great suggestion of using CallTreePrinter instead of GraphHtmlPrinter and analysing the output with kcachegrind (which is utterly amazing). Obviously your output file shouldn’t be html then…

I’ve put a CallTree output up here.

Finally… a sanctioned way of activating the screen saver.

Ever since I started managing Macs in a corporate environment, I’ve been annoyed that Apple has failed to offer a sanctioned way of locking the screen via a keyboard command. This is a reasonably common requirement in a lot of corporate deployments. Sure we can use hot corners etc, or we can use one of the sanctioned methods to activate the loginwindow via Fast User Switching, but the former isn’t for everyone, and the latter sucks because it will tear down userspace VPN/802.1x connections.

We have reasonable MCX controls to require a password for the screensaver, but nothing to actually activate it. There are a bunch of private API calls you can use to achieve this, but using private APIs makes me feel dirty.

When I first started poking at Automator again in 10.6, I was pleased to notice that we have a “Start Screen Saver” action. This means that we can save such an Automator workflow as a Service, and then assign a keyboard command to it such that we can activate it from any application.

Unfortunately this action is buggy. If you activate the workflow and wiggle the mouse around, you’ll get an error dialog after unlocking the screen.

Luckily we have another way of achieving the same goal. The System Events AppleScript dictionary contains the same functionality.

tell application "System Events"
start current screen saver
end tell

So you can simply create an Automator “Service” workflow and add a “Run AppleScript” action with the following code snippet.

on run {input, parameters}

tell app "System Events"
start current screen saver
end tell

return input
end run

Save it, and assign a hot key, and you can finally activate the screensaver from the keyboard.

Apple have documented the binary plist format

Thanks to Dave Dribin for pointing this out.


So really there’s no reason why we can’t have plistlib etc for Ruby/Python/whatever deal with binary plists on non-Mac platforms.

    magic number ("bplist")
    file format version

    variable-sized objects

    Object Formats (marker byte followed by additional info in some cases)
    null    0000 0000
    bool    0000 1000           // false
    bool    0000 1001           // true
    fill    0000 1111           // fill byte
    int 0001 nnnn   …     // # of bytes is 2^nnnn, big-endian bytes
    real    0010 nnnn   …     // # of bytes is 2^nnnn, big-endian bytes
    date    0011 0011   …     // 8 byte float follows, big-endian bytes
    data    0100 nnnn   [int]   … // nnnn is number of bytes unless 1111 then int count follows, followed by bytes
    string  0101 nnnn   [int]   … // ASCII string, nnnn is # of chars, else 1111 then int count, then bytes
    string  0110 nnnn   [int]   … // Unicode string, nnnn is # of chars, else 1111 then int count, then big-endian 2-byte uint16_t
        0111 xxxx           // unused
    uid 1000 nnnn   …     // nnnn+1 is # of bytes
        1001 xxxx           // unused
    array   1010 nnnn   [int]   objref* // nnnn is count, unless ’1111′, then int count follows
        1011 xxxx           // unused
    set 1100 nnnn   [int]   objref* // nnnn is count, unless ’1111′, then int count follows
    dict    1101 nnnn   [int]   keyref* objref* // nnnn is count, unless ’1111′, then int count follows
        1110 xxxx           // unused
        1111 xxxx           // unused

    list of ints, byte size of which is given in trailer
    — these are the byte offsets into the file
    — number of these is in the trailer

    byte size of offset ints in offset table
    byte size of object refs in arrays and dicts
    number of offsets in offset table (also is number of objects)
    element # in offset table which is top level object
    offset table offset