Canonical Voices

Posts tagged with 'puppet'

Mike Milner

In our last blog on Landscape and Puppet, we talked about using Landscape to automatically deploy Puppet on new computers. In this article we’ll dive deeper, and look at how to use Landscape as an External Node Classifier for Puppet.

In a typical Puppet configuration, all your nodes are defined in site.pp under the “node” definitions section. Each node is assigned Puppet classes by manually editing the file. When your number of nodes grows, manually managing the site.pp file becomes tedious and error prone.

An External Node Classifier (ENC) is Puppet’s way of offloading the tedious node maintenance to an external program. The interface is dead simple – puppet executes the external node classifier program with a single full node name as the only argument. The classifier just has to write a YAML blob out to stdout before exiting.

To start, let’s create a simple python ENC in /etc/puppet/landscape_enc – Don’t forget to make the file executable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
#!/usr/bin/env python
import sys
import yaml

# The node name to be classified is supplied as an argument to the script
node_name = sys.argv[1]

classes = ["basenode"]

# Output must be a YAML document
print(yaml.dump({
    "classes": classes,
    }))

 

It ignores the node name and just puts everything into the “basenode” class. Not very interesting but it’s enough to get started with Puppet.

NOTE: These examples are all using puppet 2.7 which ships on Ubuntu 12.04 Precise LTS. The ENC functionality behaves a bit differently in versions of puppet earlier than 2.65 – See http://docs.puppetlabs.com/guides/external_nodes.html for details.

To test the ENC I put together a minimal puppet configuration with two simple classes and put everything into my site.pp in /etc/puppet/manifests/site.pp

1
2
3
4
5
6
7
class basenode {
  notify {"I am a basenode!":}
}

class specialnode {
  notify {"I am a specialnode!":}
}

 

Notice that no nodes are actually defined. That is the ENC’s job. To enable the ENC you need to add two lines to your puppetmaster’s config file /etc/puppet/puppet.conf

Add these lines at the end of the “[master]” section:

1
2
node_terminus = exec
external_nodes = /etc/puppet/landscape_enc

 

You can now test that the puppetmaster is using your new, rather basic, ENC.

1
2
3
4
5
6
7
ubuntu@ubuntu:~$ sudo puppet agent --test

info: Caching catalog for ubuntu
info: Applying configuration version '1354824718'
notice: I am a basenode!
notice: /Stage[main]/Basenode/Notify[I am a basenode!]/message: defined 'message' as 'I am a basenode!'
notice: Finished catalog run in 0.06 seconds

 

As you can see, with our trivial ENC everyone is a basenode.

Now we’re going to enhance our ENC to ask Landscape to classify nodes for us.

The Landscape API
To use the Landscape API you need three pieces of information: the Landscape API endpoint URI, the user key, and the secret for your user.

To find your API credentials, log in to the Landscape web interface and click your username on the top right. Your API credentials are in the “API Access” section.

For this example, we’ll use the python API client provided with Landscape (NOTE, you must install the landscape-api package first). Here’s how to query for a computer registered with Landscape using it’s host name:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
from landscape_api.base import API

landscape_uri = "https://landscape.canonical.com/api/"
landscape_key = "43NW6OV71L32CSOPCJGX"
landscape_secret = "agBf3v267DqO8vtVRnzjseWfYdV4ueklj5a81iIT"
api = API(landscape_uri, landscape_key, landscape_secret)

api.get_computers(query="my_hostname_here")
[{u'access_group': u'server',
u'comment': u'',
u'hostname': u'appserv1',
u'id': 1,
u'last_exchange_time': None,
u'last_ping_time': u'2012-09-07T15:19:22Z',
u'reboot_required_flag': False,
u'tags': [u'lucid', u'server', u'puppet-webfarm'],
u'title': u'Application Server 1',
u'total_memory': 1024,
u'total_swap': 1024}]

 

Now if we combine that with our ENC we get the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#!/usr/bin/env python
import sys
import yaml
from landscape_api.base import API

# Create our connection to Landscape
landscape_uri = "https://landscape.canonical.com/api/"
landscape_key = "43NW6OV71L32CSOPCJGX"
landscape_secret = "agBf3v267DqO8vtVRnzjseWfYdV4ueklj5a81iIT"
api = API(landscape_uri, landscape_key, landscape_secret)

# The node name to be classified is supplied as an argument to the script
node_name = sys.argv[1]

# Ask Landscape about the computer
computers = api.get_computers(query=node_name)

# If we don't get back any computers or if we get more than one, error out.
# You could also handle this case by simply giving the node a default class.
if len(computers) != 1:
    sys.stderr.write("Only expecting one computer, instead got this: %s" % computers)
    sys.exit(1)

# Extract the tags from our computer
tags = computer[0]["tags"]

# Now here you can use whatever logic you want to convert
# tags into classes. I'm going to use any tag that starts with "puppet-"
# as a class name. I'm also going to make sure every node gets the
# "basenode" class
classes = ["basenode"]
for tag in tags:
    if tag.startswith("puppet-"):
        class_name = tag.split("-",1)[1]
        classes.append(class_name)

# Output must be a YAML document
print(yaml.dump({
    "classes": classes,
    }))

 

That’s all there is to it. Now if you tag a computer “puppet-database” in Landscape, it will automatically get the “database” class in Puppet.

You can see in the script comments that it’s very easy to customize the behaviour to match your environment. Now tag away and let Landscape and Puppet take over your world!

Read more
Michael

I’ve been playing with juju for a few months now in different contexts and I’ve really enjoyed the ease with which it allows me to think about services rather than resources.

More recently I’ve started thinking about best-practices for deploying services using juju, while still using puppet to setup individual units. As a simple experiment, I wrote a juju charm to deploy an irssi service [1] to dig around. Here’s what I’ve found so far [2]. The first is kind of obvious, but worth mentioning:

Install hooks can be trivial:

#!/bin/bash
sudo apt-get -y install puppet

juju-log "Initialising machine state."
puppet apply $PWD/hooks/initial_state.pp

Normally the corresponding manifest (see initial_state.pp) would be a little more complicated, but in this example it’s hardly worth mentioning.

Juju config changes can utilise Puppet’s Facter infrastructure:

This enables juju config options to be passed through to puppet, so that config-changed hooks can be equally simple:

#!/bin/bash
juju-log "Getting config options"
username=`config-get username`
public_key=`config-get public_key`

juju-log "Configuring irssi for user"
# We specify custom facts so that they're accessible in the manifest.
FACTER_username=$username FACTER_public_key=$public_key puppet apply $PWD/hooks/configured_state.pp

In this example, it is the configured state manifest that is more interesting (see configured_state.pp). It adds the user to the system, sets up byobu with an irssi window ready to go, and adds the given public ssh key enabling the user to login.

The same would go for other juju hooks (db-relation-changed etc.), which is quite neat – getting the best of both worlds: the charm user can still think in terms of deploying services, while the charm author can use puppets declarative syntax to define the machine states.

Next up: I hope to experiment with an optional puppet master for a real project (something simple like the Ubuntu App directory), so that

  1. a project can be deployed without the (probably private) puppet-master to create a close-to-production environment, while
  2. configuring a puppet-master in the juju config would enable production deploys (or deploys of exact replicas of production to a separate environment for testing).

If you’re interested in seeing the simple irssi charm, the following 2min video demos:

# Deploy an irssi service
$ juju deploy --repository=/home/ubuntu/mycharms  local:oneiric/irssi
# Configure it so a user can login
$ juju set irssi username=michael public_key=AAAA...
# Login to find irssi already up and running in a byobu window
$ ssh michael@new.ip.address

and the code is on Launchpad.

[1] Yes, irssi is not particularly useful as a juju service (as I don’t want multiple units, or relating it to other services etc.), but it suited my purposes for a simple experiment that also automates something I can use for working in the cloud.

[2] I’m not a puppet or juju expert, so if you’ve got any comments or improvements, don’t hesitate.


Filed under: juju, puppet, ubuntu

Read more

Now it’s not deja vu. This happened to me before. My other best friend is now also moving to California this month to work for a large company who is deploying tons of Puppet. It’s a tremendous opportunity for him, just like the last guy.

Best of luck to him, it’s always good to see projects like Puppet flourish and for Ken to have an opportunity like this. So as a goodbye/good luck we went to go see the Wings play the Sharks at the Joe. Unfortunately for him his home team will now be the San Jose Sharks, heh.

Here’s a picture of us at the game. (He’s the abandoner on the left.)

And my dad was in town as well, and hadn’t seen the Wings in like 15 years so he tagged along too.

Wings lost 2-5. :(

Read more