This has been the big project I’ve been working on since I left Google for the life of being a product manager at a startup software company. Lack of sleep, worrying about names, trying impose order on the birthing process…. it’s not that unlike real babies.

I’m proud of Puppet Enterprise.

Puppet is an awesomely powerful tool, but it does take some effort to get a scalable deployment going. It’s not a lot of work if you’re a reasonably experienced sysadmin, but some people simply don’t have the time.

We’ve launched with support for RHEL, CentOS, OEL, Debian and Ubuntu, and I’m particularly looking forward to our upcoming Mac OS X version.  I may have left the MacEnterprise.org world behind, but all my Mac IT peeps still hold a special place in my heart.

There’s barely time to take a breath now 1.0 is out, as I’m heading off to FOSDEM in a couple of days to run the configuration management devroom with James Turnbull.

I’m planning to conduct some in-depth quality control inspections of Belgian beer while I’m in Brussels. Suggestions for inspection locations are more than welcome.

It’s been a crazy 4 years. Helping set up the largest Mac deployment I’ve ever dealt with, getting into Puppet, learning a lot of Python, discovering how robust you can make bash, moving to the Linux teams, learning from a lot of awfully smart people, and dealing with the fallout from the Operation Aurora attacks earlier this year… it’s hard to believe it’s only been 4 years in some ways.

It felt right to put something up here about this, but don’t expect any earth shattering revelations or horror stories.

I thoroughly enjoyed working at Google. I loved working with so many talented people, and I loved getting to work on infrastructure at scale that was developed by smart people.

However, the time was right to leave. The endless suburbia of the Bay Area was starting to get to us, and I’ve been very close to the Puppet Labs folks for a long time now, so when they offered me the position of Product Manager, and a chance to finally put my money where my mouth is about Puppet… well I couldn’t turn it down.

We have a lot of work to do at Puppet Labs, but it’s exciting work, as I honestly feel that we have a great chance of helping to fundamentally improve our industry. What more could I ask for out of a job?

http://glarizza.posterous.com/using-crankd-to-react-to-network-events

This is exactly what Chris and I wrote it for initially, so it’s great to see this info out there.

Now I have somewhere to point Greg if he bugs me about it again. :)

I just added the pymacds module to the pymacadmin project.

To install the eggs for Python2.5 (OS X 10.5 only comes with 2.5) or Python2.6 (OS X 10.6 comes with both), run something like:


curl -O http://pymacadmin.googlecode.com/files/pymacds-0.3-py2.5.egg
sudo easy_install-2.5 pymacds-0.3-py2.5.egg


or


curl -O http://pymacadmin.googlecode.com/files/pymacds-0.3-py2.6.egg
sudo easy_install-2.6 pymacds-0.3-py2.6.egg


Once you’ve done this, you can use it like:


nigelk$ python2.5
Python 2.5.4 (r254:67916, Feb 11 2010, 00:50:55)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymacds
>>> pymacds.UserAttribute('nigelk', 'NFSHomeDirectory')
[u'/Users/nigelk']
>>> pymacds.GetSearchNodes()
['/Local/Default', '/BSD/local', '/LDAPv3/my.ldap.node']
>>> pymacds.GetContactsNodes()
['/LDAPv3/my.ldap.node']
>>>


There are a few other useful methods, listing them here rather than going through them one by one. We’re particularly fond of the Ensure* methods for triggering DirectoryService node addition/removal on network events via crankd.


FlushCache():
GetSearchNodes():
GetContactsNodes():
AddNodeToSearchPath(node):
AddNodeToContactsPath(node):
DeleteNodeFromSearchPath(node):
DeleteNodeFromContactsPath(node):
EnsureSearchNodePresent(node):
EnsureSearchNodeAbsent(node):
EnsureContactsNodePresent(node):
EnsureContactsNodeAbsent(node):
DSQuery(dstype, objectname, attribute=None):
DSSet(dstype, objectname, attribute=None, value=None):
DSDelete(dstype, objectname, attribute=None, value=None):
UserAttribute(username, attribute):
GroupAttribute(groupname, attribute):
AddUserToLocalGroup(username, group):
RemoveUserFromLocalGroup(username, group):


Code contributions welcome.

If you’re talking about replacing shell scripts, all of these are pretty much equivalent, but I don’t really do Perl for no particular reason, I’m a big fan of Python for general purpose work just because of the rich module system, and at least “dozens of engineers” use it at work..

The reason we generally write shell scripts is because we want to execute a bunch of external processes in an automated way.

In this area you can whip something basic up fastest in shell, yes, but at some point you’re going to have to repay that technical debt if you need to get past a certain point.

Besides, it’s not like shell scripting always stays simple and easy… and the overhead of moving to a more powerful language isn’t that huge.

If I’m sure the scope of a script will be small, or I don’t have the option of moving to structured data format for input and output with external commands, or I don’t have to futz around with arrays, I’ll stick with bash.

But, if I want to work with dictionary objects or talk in a protocol like LDAP or model things as objects, or need complex handling and passing around of stdin/stdout/exit statuses, or know some module handles lots of edge cases for me, I’ll move to Python or Ruby. I quite like both, but feel that Python is more utilitarian, and simply due to whitespace enforcement and extensive linters is a good fit for code that may need to be picked up and understood quickly by a co-worker.

When it comes to getting started with Python, I still suggest Dive into Python for people. Just flipping through Chapters 1-3 equips you with an awful lot.

Anyway, some people think Python is hard, but it’s not really. I think of Python as being extremely utilitarian, which makes it a great fit for sysadmin work.

Someone posted on the MacEnterprise list a question about working out what PPD was in use by each printer on the system in OS X, and someone gave a good shell example using the usual suspects of for, grep and awk.

There’s nothing really wrong with doing this, but I’ve come to distrust parsing non-structured output if I need to keep this solution working across many multiple versions of operating systems, or even between OS X major versions given Apple’s history. One of the big advantages of moving to Python is being able to parse and manipulate property lists, which can get really painful in shell. You end up writing out lots of temp files or giving up on dealing with error conditions.

Anyway, so lets see what it’s like executing a command in Python with the subprocess module. We’re asking System Profiler for printer info, and telling it to return the output in XML plist format.

To skim through those lines, we’re

• setting the python shebang
• importing the subprocess module
• defining a list called ‘command’ to store the command we want to run
• creating a subprocess Popen object to run our command called ‘task’
• setting standard out and standard error to go to our own pipes
• getting standard out and standard error from the task.
• printing standard out

This really isn’t much work, and if you really only wanted this functionality, there are other convenience functions that you can use to make this even shorter, or write your own convenience function.

But we have all sorts of options now.

We can send standard input to the task:

We can ask what the exit status is easily:

and if we get None we know the process hasn’t terminated yet.

And if we want to do something a bit more complicated we can search through the structured data plist output easily like:

This is easier to extend than the standard for/grep/awk/sed equivalents tend to be, and much easier for a co-worker to pick up and understand. It comes close to documenting itself, just needs some comments about the structure of Apple’s output, and some try/except blocks in case the output is malformed or does change.

MacEnterprise is planning to partner with various other groups to get
a Mac IT focused conference started.

This is all very much up in the air, and at this stage we’re seeking
input as to how the community would like this conference to be
organized.

At this stage, we would like you to provide input on Google Moderator
as to ideas for the conference. You can submit ideas as well as vote
on other ideas here:

http://goo.gl/mod/4COQ

Additionally, there is some discussion going on on Twitter, under the
#MacITConference hashtag.

http://twitter.com/#search?q=%23MacITConference

Once we get a little bit better idea of the structure, we’ll be
calling for speakers and looking for sponsorship partners.

Unfortunately I’ve yet to find a smoking gun, but here’s a reasonably simple way to produce profiles of puppetmasterd.

• Install ruby-prof from gems.
• Stop any apache/mongrel/nginx instances of puppetmasterd you may have running
• Edit /usr/sbin/puppetmasterd and replace the last few lines with:
  require ‘rubygems’
  require ‘ruby-prof’

  result = RubyProf.profile do
    require ‘puppet/application/puppetmasterd’
    Puppet::Application[:puppetmasterd].run
  end

  printer = RubyProf::GraphHtmlPrinter.new(result)

  File.open(‘/tmp/ruby-profile.html’, ‘w’) do |file|
    printer.print(file, {:min_percent => 10, :print_file => true})
  end

• Start a webrick puppetmasterd with –no-daemonize
• Do a client run against it
• Hit Ctrl-C to interrupt your puppetmasterd
• wait for the html output to be generated

It’s worth filtering the min_percent value. Without it, I ended up with 300+MB HTML files with no images that took my dev server a long time to write to disk. With it, I end up with a couple of megs.

You can see an example output at:
http://www.explanatorygap.net/crap/ruby-profile.html

with the interesting thread being at:
http://www.explanatorygap.net/crap/ruby-profile.html#70121801903160

Interpretation suggestions welcome :)

Edit:

Brice had a great suggestion of using CallTreePrinter instead of GraphHtmlPrinter and analysing the output with kcachegrind (which is utterly amazing). Obviously your output file shouldn’t be html then…

I’ve put a CallTree output up here.

We have reasonable MCX controls to require a password for the screensaver, but nothing to actually activate it. There are a bunch of private API calls you can use to achieve this, but using private APIs makes me feel dirty.

When I first started poking at Automator again in 10.6, I was pleased to notice that we have a “Start Screen Saver” action. This means that we can save such an Automator workflow as a Service, and then assign a keyboard command to it such that we can activate it from any application.

Unfortunately this action is buggy. If you activate the workflow and wiggle the mouse around, you’ll get an error dialog after unlocking the screen.

Luckily we have another way of achieving the same goal. The System Events AppleScript dictionary contains the same functionality.


tell application "System Events"
start current screen saver
end tell


So you can simply create an Automator “Service” workflow and add a “Run AppleScript” action with the following code snippet.


on run {input, parameters}

return input
end run


Save it, and assign a hot key, and you can finally activate the screensaver from the keyboard.

In http://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c

So really there’s no reason why we can’t have plistlib etc for Ruby/Python/whatever deal with binary plists on non-Mac platforms.

It will appear here when done.
http://packages.debian.org/sid/puppet

Instructions for building yourself….

$ git clone git://git.debian.org/pkg-puppet/puppet.git
$ cd puppet
$ git-buildpackage --git-upstream-branch=origin/upstream