So I’m not hugely happy with the CPU consumption of puppetmasterd under heavy load, and so I’ve been trying to work out where the bottlenecks lie.

Unfortunately I’ve yet to find a smoking gun, but here’s a reasonably simple way to produce profiles of puppetmasterd.

• Install ruby-prof from gems.
• Stop any apache/mongrel/nginx instances of puppetmasterd you may have running
• Edit /usr/sbin/puppetmasterd and replace the last few lines with:
  require ‘rubygems’
  require ‘ruby-prof’

  result = RubyProf.profile do
    require ‘puppet/application/puppetmasterd’
    Puppet::Application[:puppetmasterd].run
  end

  printer = RubyProf::GraphHtmlPrinter.new(result)

  File.open(‘/tmp/ruby-profile.html’, ‘w’) do |file|
    printer.print(file, {:min_percent => 10, :print_file => true})
  end

• Start a webrick puppetmasterd with –no-daemonize
• Do a client run against it
• Hit Ctrl-C to interrupt your puppetmasterd
• wait for the html output to be generated

It’s worth filtering the min_percent value. Without it, I ended up with 300+MB HTML files with no images that took my dev server a long time to write to disk. With it, I end up with a couple of megs.

You can see an example output at:
http://www.explanatorygap.net/crap/ruby-profile.html

with the interesting thread being at:
http://www.explanatorygap.net/crap/ruby-profile.html#70121801903160

Interpretation suggestions welcome :)

Edit:

Brice had a great suggestion of using CallTreePrinter instead of GraphHtmlPrinter and analysing the output with kcachegrind (which is utterly amazing). Obviously your output file shouldn’t be html then…

I’ve put a CallTree output up here.

We’ve uploaded the 0.25.1 debs, but due to this work, it might take a little while before they appear.
http://blog.ganneff.de/blog/2009/10/27/debian-ftpmaster-meeting.html

It will appear here when done.
http://packages.debian.org/sid/puppet

Instructions for building yourself….

$ git clone git://git.debian.org/pkg-puppet/puppet.git
$ cd puppet
$ git-buildpackage --git-upstream-branch=origin/upstream


If you’re a Mac IT person, and you don’t know about Greg Neagle’s Managing OS X blog, you need to fix that situation now.

One of the reasons Greg is so awesome in our field is that he’s eminently pragmatic, with enough hacker mentality to make sure he simply gets the job done with a minimum of fuss. His recent post on the trials and tribulations of working with the Adobe Enterprise Deployment Kit is a great example.

Not only is he trying to come up with something flexible enough to actually use efficiently, he’s dug into the innards and explained exactly what’s going on.

I talked to a few people at Puppet Camp last week about large scale Mac management, and everyone seemed really excited about the Munki Project, which is all Greg’s work so far. Basically the idea is to provide OS X with an actual repository for package management, using native Mac packages, and attempting to reuse vendor packages as much as is feasible.

If no-one else does it, I’ll end up putting together a munki type and provider for Puppet. I’m really looking forward to being able to simply do stuff like:

package { "iWork":
  ensure => latest,
}

just like other operating systems, letting the repository handle dependencies. The way it should be….

This really could be one of the most important community contributions to large scale Mac management in the history of OS X in my opinion.

I’ve submitted a diff to update facter in MacPorts to 1.5.7, so it should be available soon.

Note that I’ve set the maintainer for both Puppet and Facter in MacPorts to ‘openmaintainer’. This means that I accept patches from anyone, and it’s really quite trivial to update either of them, as is the case with the vast majority of Portfiles.

The process goes something like:

$ sudo port selfupdate (to get the newest versions)
$ mkdir /tmp/facter
$ cd /tmp/facter
$ cp /opt/local/var/macports/sources/rsync.macports.org/release/ports/sysutils/facter/Portfile .
$ cp Portfile Portfile.orig
(edit the port file to change version from 1.5.6 to 1.5.7)
$ port -v checksum (this will print out the expected and obtained checksums. Use this info to update the ‘checksums’ component of the Portfile)
$ port -v checksum (this should return happily now)
$ sudo port -v install (verify that the port is installed correctly)
$ diff -u Portfile Portfile.orig > Portfile-facter.diff (submit an update ticket on the MacPorts Trac site with the diff attached)

This has been flowing all over the #puppetcamp twitter tag, but it’s worth repeating.

“Think of the complexity in your environment as a form of technical debt that you’re going to have to pay down” – Paul Nasrat

This is so awesomely pithy you just know he’s a bloody Pom.

(England 3/83 in the Champions Trophy semi-final as of right now…)

Puppet Camp is on today and tomorrow.

It’s already exciting being in a room full of involved sysadmins who are concerned with making our jobs better and thinking about how the place our field will be in in the next few years…

It’s always good to put faces to IRC handles too :)

Already had a great talk from Ohad Levy on The Foreman and his infrastructure. I’m excited about The Foreman, even if we don’t end up using it at Google.

In case you’ve been wondering where the debs are for Puppet 0.25.x, we’ve decided to wait until 0.25.1 to publish it to Debian unstable.

The good news is that we’ve set things up in our Alioth git repository so that you can use git-buildpackage to build 0.25.x debs. We’ll publish an article on the Debian wiki and link to it from the Puppet wiki that will describe how to build debs for any 0.25.x branch of puppet using git-buildpackage, which should make it a lot easier for people to work on the bleeding edge.

If you’re already familiar with git-buildpackage, you can work against the anonymous repository  now at: git://git.debian.org/pkg-puppet/puppet.git

The Mac packages for Puppet 0.25.0 are done:

https://sites.google.com/a/explanatorygap.net/puppet/

and the Portfile for MacPorts is also now available.