Puppet 2.7 Cookbook

By John Arundel

This book is written in a Cookbook style showing you how to set up and expand your Puppet infrastructure. It progresses through detailed information on the language and features, external tools, reporting, monitoring, and troubleshooting, and concludes with many specific recipes for managing popular applications. The book assumes that the reader already has a working Puppet installation and perhaps has written some basic manifests or adapted some published modules. It also requires some experience of Linux systems administration, including familiarity with the command line, file system, and text editing. No programming experience is required.

• Language: English
• Publisher: Packt Publishing
• Release date: Oct 24, 2011
• ISBN: 9781849515399

John Arundel

John Arundel is a well-known Go teacher and mentor. He has been writing software for 40 years and thinks he's starting to figure out how to do it. You can find out more at bitfieldconsulting.com. He lives in a fairytale cottage in Cornwall, England, surrounded by woods, wildlife, and a slowly deepening silence.

Book preview

Table of Contents

Puppet 2.7 Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Support files, eBooks, discount offers and more

Why Subscribe?

Free Access for Packt account holders

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Errata

Piracy

Questions

1. Puppet Infrastructure

Using version control

Getting ready

How to do it...

How it works...

There's more...

See also

Using commit hooks

How to do it…

How it works…

There's more…

See also

Deploying changes with Rake

Getting ready

How to do it...

How it works...

There's more...

See also

Configuring Puppet's file server

How to do it...

How it works...

There's more...

See also

Running Puppet from cron

How to do it...

How it works...

There's more...

See also

Using autosign

How to do it...

How it works...

See also

Pre-signing certificates

How to do it...

See also

Retrieving files from Puppet's filebucket

How to do it...

How it works...

There's more...

Scaling Puppet using Passenger

Getting ready

How to do it...

How it works...

There's more...

See also

Creating decentralized Puppet architecture

Getting ready

How to do it...

How it works...

There's more...

See also

2. Monitoring, Reporting, and Troubleshooting

Generating reports

How to do it…

How it works…

There's more…

Enabling reports on the command line

Logging Puppet messages to syslog

See also

E-mailing log messages containing specific tags

How to do it…

How it works…

There's more…

What are tags?

Specifying multiple tags, or excluding tags

Sending reports to multiple e-mail addresses

See also

Creating graphical reports

Getting ready

How to do it…

How it works…

There's more…

See also

Producing automatic HTML documentation

How to do it…

How it works…

There's more…

Drawing dependency graphs

Getting ready…

How to do it…

How it works…

There's more…

Testing your Puppet manifests

Getting ready

How to do it…

How it works…

There's more…

Doing a dry run

How to do it…

How it works…

There's more…

See also

Detecting compilation errors

How to do it…

How it works…

Understanding Puppet errors

How to do it…

Logging command output

How to do it…

How it works…

There's more…

Logging debug messages

How to do it…

How it works…

There's more…

Printing out variable values

Printing the full resource path

Logging messages on the Puppetmaster

Inspecting configuration settings

How to do it…

How it works…

Using tags

How to do it...

There's more…

Using run stages

How to do it…

How it works…

There's more…

Using environments

How to do it…

How it works…

There's more…

See also

3. Puppet Language and Style

Using community Puppet style

How to do it…

There's more…

Using modules

How to do it…

How it works…

There's more...

Templates

Facts, functions, types, and providers

puppet-module

Third-party modules

Module organization

See also

Using standard naming conventions

How to do it…

There's more…

Using embedded Ruby

How to do it…

How it works…

See also

Writing manifests in pure Ruby

How to do it…

How it works…

There's more…

Variables

Documentation

Iterating over multiple items

How to do it…

How it works…

There's more…

Hashes

Creating arrays with the split function

Writing powerful conditional statements

How to do it…

How it works…

There's more…

elsif

Comparisons

Combining expressions

See also

Using regular expressions in if statements

How to do it…

How it works…

There's more…

Capturing patterns

Regular expression syntax

See also

Using selectors and case statements

How to do it…

How it works…

There's more…

Regular expressions

Defaults

Testing whether values are contained in strings

How to do it…

There's more…

Using regular expression substitutions

How to do it…

How it works…

There's more...

See also

4. Writing Better Manifests

Using arrays of resources

How to do it…

How it works…

See also

Using define resources

How to do it…

How it works…

There's more…

Using dependencies

How to do it…

How it works…

There's more…

Using node inheritance

How to do it…

How it works…

There's more…

See also

Using class inheritance and overriding

Getting ready…

How to do it…

How it works…

There's more…

Undefining parameters

Adding extra values using the +> operator

Disabling resources

See also

Passing parameters to classes

How to do it…

How it works…

There's more…

See also

Writing reusable, cross-platform manifests

How to do it…

How it works...

There's more…

See also

Getting information about the environment

How to do it…

How it works…

There's more…

See also

Importing dynamic information

Getting ready…

How to do it…

How it works…

There's more…

See also

Importing data from CSV files

Getting ready…

How to do it…

How it works…

There's more…

See also

Passing arguments to shell commands

How to do it…

How it works…

5. Working with Files and Packages

Making quick edits to config files

How to do it…

How it works…

There's more…

See also

Using Augeas to automatically edit config files

Getting ready…

How to do it…

How it works…

There's more…

Building config files using snippets

How to do it…

How it works…

There's more…

See also

Using ERB templates

How to do it…

How it works…

There's more…

See also

Using array iteration in templates

How to do it…

How it works…

There's more…

See also

Installing packages from a third-party repository

How to do it…

How it works…

There's more...

See also

Setting up an APT package repository

Getting ready…

How to do it…

How it works…

There's more…

Adding packages

Configuring nodes to use the repository

Signing your packages

Setting up a gem repository

How to do it…

How it works…

There's more…

Adding gems

Using the gem repo

Building packages automatically from source

How to do it…

How it works…

There's more…

Comparing package versions

How to do it…

How it works…

6. Users and Virtual Resources

Using virtual resources

How to do it…

How it works…

There's more…

See also

Managing users with virtual resources

How to do it…

How it works…

See also

Managing users' SSH access

How to do it…

How it works…

There's more…

Managing users' customization files

How to do it…

How it works…

See also

Efficiently distributing cron jobs

How to do it…

How it works…

There's more…

See also

Running a command when a file is updated

Getting ready…

How to do it…

How it works…

There's more…

Using host resources

How to do it…

How it works...

There's more...

Using multiple file sources

How to do it…

How it works…

There's more...

See also

Distributing directory trees

How to do it…

How it works…

There's more…

Cleaning up old files

How to do it…

How it works…

There's more…

Using schedules with resources

How to do it…

How it works…

There's more…

Auditing resources

How to do it…

How it works…

There's more…

See also

Temporarily disabling resources

How to do it…

How it works…

Managing timezones

How to do it…

There's more…

7. Applications

Managing Apache servers

How to do it...

There's more...

Creating Apache virtual hosts

How to do it…

How it works…

There's more…

See also

 Creating Nginx virtual hosts

Getting ready…

How to do it…

How it works…

There's more…

See also

Creating MySQL databases and users

Getting ready…

How to do it…

How it works…

There's more…

Managing Drupal sites

Getting ready…

How to do it…

How it works…

There's more…

Managing Rails applications

How to do it…

How it works…

Nginx and Passenger

Rails

There's more…

RVM

Log rotation

Databases

SSL certificates

8. Servers and Cloud Infrastructure

Deploying a Nagios monitoring server

Getting ready…

How to do it…

How it works...

There's more…

Building high-availability services using Heartbeat

How to do it…

How it works…

There's more…

Managing NFS servers and file shares

How to do it…

How it works…

There's more…

Using HAProxy to load-balance multiple web servers

How to do it…

How it works…

There's more…

Managing firewalls with iptables

Getting ready…

How to do it…

How it works…

There's more…

Managing EC2 instances

Getting ready…

How to do it…

How it works…

There's more...

See also

Managing virtual machines with Vagrant

How to do it…

How it works…

There's more…

9. External Tools and the Puppet Ecosystem

Creating custom Facter facts

Getting ready...

How to do it…

How it works...

There's more...

Executing commands before and after Puppet runs

How to do it…

There's more…

Generating manifests from shell sessions

Getting ready…

How to do it…

How it works…

There's more…

Generating manifests from a running system

How to do it…

There's more…

Using Puppet Dashboard

Getting ready…

How to do it…

How it works…

There's more…

See also

Using Foreman

Getting ready…

How to do it…

There's more…

Using MCollective

 Getting ready...

How to do it…

How it works…

There's more…

Installing an MCollective plugin

Using public modules

Getting ready...

How to do it…

How it works…

There's more…

Using an external node classifier

Getting ready…

How to do it…

How it works…

There's more…

Creating your own resource types

Getting ready…

How to do it…

How it works…

There's more…

Documentation

Validation

Creating your own providers

Getting ready…

How to do it…

How it works…

There's more…

Index

Puppet 2.7 Cookbook

---

Puppet 2.7 Cookbook

Copyright © 2011 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

First published: October 2011

Production Reference: 1171011

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-84951-538-2

www.packtpub.com

Cover Image by Sujay Gawand (<[e-mail]>)

Credits

Author

John Arundel

Reviewers

Mark Phillips

Eric Stonfer

Acquisition Editors

Chaitanya Apte

Kartikey Pandey

Development Editor

Alina Lewis

Technical Editors

Priyanka S

Ankita Shashi

Project Coordinator

Michelle Quadros

Proofreader

Matthew Humphries

Indexer

Monica Ajmera

Graphics

Valentina Joseph D'silva

Production Coordinator

Prachali Bhiwandkar

Cover Work

Prachali Bhiwandkar

About the Author

John Arundel is a consultant engineer who helps people build better infrastructure. He uses automation and configuration management to make computer systems cheaper, faster, and more reliable. Formerly a senior enterprise systems engineer in the hosting division of US telco Verizon, he now runs his own company, Bitfield Consulting, and says he has never worked so hard in his life, or for less money.

Over the years John has worked with clients in the advertising and media industry, software, finance, retail, logistics, and even the emergency services, advising on architecture, automation, security, backups, resilience, performance, capacity planning, and regulatory compliance. He has been a member of the Puppet community since its earliest days, and organizes regular local sysadmin meetups and social events.

John holds a B.Sc.(Hons) in Computer Science, with a research interest in kernel resource scheduler design, and is a certified Sun Solaris administrator, LPI (Linux Professional Institute) graduate, and a member of the British Computer Society (MBCS). He is security-cleared to work on computer systems for the UK nuclear industry, which is probably nothing to worry about.

He has also worked as a software developer, both professionally and for the fun of it, contributing to several open source projects, and building a high-performance research chess engine. He blogs regularly at http://bitfieldconsulting.com on Puppet and system administration topics, is usually to be found on Twitter (@bitfield) complaining about things, and often speaks at technical user groups and conferences.

In his negligible spare time, John enjoys repairing Land Rovers, playing Go, and barbecuing. He lives in London and Cornwall.

My thanks go to Luke Kanies and the team at Puppet Labs; also to Ken Barber, Lindsay Holmwood, Gary Larizza, Stephen Nelson-Smith, R.I. Pienaar, Julian Simpson, Jordan Sissel, Cosimo Streppone, James Turnbull, and Dean Wilson, who all provided valuable contributions to the book, whether they know it or not; and for their brave self-sacrifice in the cause of proofreading, Ian Chilton, Kris Buytaert, Stefan Goethals, and Martin Brooks. A special mention goes to the regulars of channel #puppet, who often helped out when things didn't work the way they were supposed to, which was virtually all the time.

About the Reviewers

Mark Philips has had a varied career spanning Motor Manufacturer, Internet, Telco, and Finance industries over the last 17 years. Engineering for UNIX estates from a handful of hosts through to many thousands, Mark has strived to automate anything and everything that had to be carried out more than once. Discovering Puppet in early 2007 was a boon to achieving his idea of systems nirvana—simple centralized and automated configuration management.

Mark runs an IT consultancy company, VNTX Limited, specializing in UNIX installation, integration, automation, and performance tuning.

When he's not in front of a computer, Mark can be found out riding one of his bicycles—training for a race, or boring his ever patient wife talking about cycling.

Eric Stonfer is a 10 year veteran of systems administration, with an emphasis on automation and configuration systems, and has been using Puppet to manage thousands of servers for over 3 years. In his spare time Eric is an avid home brewer.

www.PacktPub.com

Support files, eBooks, discount offers and more

You might want to visit www.PacktPub.com for support files and downloads related to your book.

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks.

http://PacktLib.PacktPub.com

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can access, read and search across Packt's entire library of books. 

Why Subscribe?

Fully searchable across every book published by Packt

Copy and paste, print and bookmark content

On demand and accessible via web browser

Free Access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view nine entirely free books. Simply use your login credentials for immediate access.

Preface

A revolution is coming to IT operations. Configuration management tools can build servers in seconds and automate your entire network. Tools like Puppet are essential to take full advantage of the power of cloud computing, and build reliable, scalable, secure, and high-performance systems.

This book takes you beyond the basics and explores the full power of Puppet, showing you in detail how to tackle a variety of real-world problems and applications. At every step, it shows you exactly what commands you need to type and includes complete code samples for every recipe.

It takes the reader from rudimentary knowledge of Puppet to a more complete and expert understanding of Puppet's latest and most advanced features, community best practices, writing great manifests, scaling and performance, and how to extend Puppet by adding your own providers and resources.

This book also includes real examples from production systems and techniques that are in use in some of the world's largest Puppet installations, including a distributed Puppet architecture and a high-performance Puppetmaster solution using Apache and Passenger.

Explore the power of Puppet with this practical guide to the world's most popular configuration management system.

What this book covers

Chapter 1, Puppet Infrastructure introduces some key techniques for managing your Puppet server and manifests, including version control, automated deployment, file serving, pre-signing and autosigning certificates, scaling with Passenger, and a distributed decentralized Puppet architecture using Git.

Chapter 2, Monitoring, Reporting, and Troubleshooting covers ways that Puppet can report information about what it's doing, and the status of your systems. This includes graphical and e-mail reports, log and debug messages, dependency graphing, testing and dry-running your manifests, using tags, run stages, and environments, and a guide to some of Puppet's more common error messages.

Chapter 3, Puppet Language and Style will show you examples of good programming style in Puppet and language constructs that can help you keep your code concise and readable, including conditionals, selectors, case statements, arrays, and regular expressions.

Chapter 4, Writing Better Manifests takes you through structuring your Puppet manifests using node and class inheritance, resource dependencies, and parameterized classes. You'll also see how to get data in and out of Puppet from the environment using CSV files and shell scripts.

Chapter 5, Working with Files and Packages covers powerful techniques for managing config files, including ERB templates, generating files from snippets, and using the Augeas tool. You'll also see how to use Puppet to install packages from APT repositories, and how to set up your own APT and Gem repositories.

Chapter 6, Users and Virtual Resources explains how virtual resources can help you manage different combinations of users and packages on different machines, and shows you how to use Puppet's resource scheduling and auditing features.

Chapter 7, Applications focuses on some specific applications that you may need to manage with Puppet, including complete recipes for Apache and Nginx, MySQL, Drupal, and Rails.

Chapter 8, Servers and Cloud Infrastructure extends the power of Puppet to managing virtual machines, both in the cloud and on your desktop, with recipes for Vagrant and EC2 instances. It also shows you how to set up a Nagios monitoring server, load balancing with HAProxy, firewalls with iptables, network filesystems with NFS, and high-availability services with Heartbeat.

Chapter 9, External Tools and the Puppet Ecosystem looks at the tools that have grown up around Puppet and help you integrate it with the rest of your network, including Puppet Dashboard, Foreman, and MCollective. It also introduces you to some advanced topics including writing your own resource types, providers, and external node classifiers.

What you need for this book

To run the examples in this book, you will need a computer with Ubuntu Linux 10.04 and Puppet installed, and an Internet connection. Though not strictly necessary, I also recommend an espresso machine or some other form of caffeinated beverage dispenser.

Who this book is for

The book assumes that the reader already has a working Puppet installation and perhaps has written some basic manifests or adapted some published modules. It also requires some experience of Linux systems administration, including familiarity with the command line, file system, and text editing. No programming experience is required.

Conventions

In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.

Code words in text are shown as follows: You'll need a Puppetmaster and a set of existing manifests in /etc/puppet.

A block of code is set as follows:

#!/bin/sh

 

syntax_errors=0

error_msg=$(mktemp /tmp/error_msg.XXXXXX)

 

if git rev-parse --quiet --verify HEAD > /dev/null

then

    against=HEAD

Any command-line input or output is written as follows:

# puppet parser validate/etc/puppet/manifests/site.pp err: Could not parse for environment production: Syntax error at end of file at /etc/puppet/manifests/site.pp:3

New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: clicking the Next button moves you to the next screen.

Note

Warnings or important notes appear in a box like this.

Tip

Tips and tricks appear like this.

Reader feedback

Feedback from our readers is