Securing /etc/puppet on CentOS

I’ve recently delved in to the world of Puppet to manage some CentOS servers. In the process I noticed something. The /etc/puppet directory is owned by root:root but puppet runs as the user puppet. What does this mean? A couple of things:

  • To edit the manifests or modules I either have to be root or constantly be typing sudo (annoying).
  • For the puppetmaster process, which runs as puppet:puppet to access the files, the manifest and modules must be world readable.  This means a lot of information is visible to the world, encrypted or not.
  • I can’t use my favorite editor to edit files over ssh. (I know, a personal gripe, but valid in my books.)

So I’m trying an experiment that I hope will secure the data a bit more and make editing the files more hastle free.

  • Recursively changed the group of /etc/puppet to puppet.
  • Put myself in the puppet group.  I can now edit the files without being root.  (See newgrp(1).)
  • I’ll slowly begin to set the Other permission bits to 0, hiding the files and their contents from prying eyes.

 

 

 

Test your Docker image builds on multiple Linux distributions

Test your Docker image builds on multiple Linux distributions.

Why? It seems things behave differently on different distos. Here is a situation I just ran into.

I have been working on a new Docker image for an up coming post. The post uses CentOS and I generally use CentOS on my servers so I naturally built the image on a CentOS host. Once everything was way I wanted it I committed the changes to Github and had Dockerhub pull and build the image. To my surprise the build failed with the error

Could not find 'which' command, make sure it's available first before continuing installation.

I went back to my CentOS host and built the image again.  No errors.  On a hunch, I created an Ubuntu VM to build the image. Bingo! While the image built cleanly on CentOS, under Ubuntu it would fail with the error above.

While the fix was as simple as explicitly installing the which package as part of the build process, it showed me two things.

  1. Don’t assume your images build on all Linux distributions.
  2. Don’t assume Docker behaves the same on all Linux Distributions.

Happy image building!

Puppet-lint Plugins List

Based on the Puppet-lint Plugins list available at the Puppet Community site, I’ve added the gem command line and Gemfile commands for easy installation.

absolute_classname

  • Check relative class name inclusions.
  • gem install puppet-lint-absolute_classname-check

absolute_template

  • Check if paths to the template() function are relative.
  • gem install puppet-lint-absolute_template_path

alias

  • Check for alias parameters in resources.
  • gem install puppet-lint-alias-check
  • gem 'puppet-lint-alias-check', :require => false

appends

  • Check that the appends operator (+=) is not used (removed in Puppet 4.0.0).
  • gem install puppet-lint-appends-check
  • gem 'puppet-lint-appends-check', :require => false

classes_and_types_beginning_with_digits

  • Check for types and class names that begin with digits.
  • gem install puppet-lint-classes_and_types_beginning_with_digits-check
  • gem 'puppet-lint-classes_and_types_beginning_with_digits-check', :require => false

empty_string

  • Check for variables assigned to the empty string.
  • gem install puppet-lint-classes_and_types_beginning_with_digits-check
  • gem 'puppet-lint-empty_string-check', :require => false

file_ensure

  • Check the ensure attribute on file resources.
  • gem install puppet-lint-file_ensure-check
  • gem 'puppet-lint-file_ensure-check', :require => false

file_source_rights

  • Check file rights when providing a source.
  • gem install puppet-lint-file_source_rights-check
  • gem 'puppet-lint-file_source_rights-check', :require => false

fileserver

  • Check if puppet:/// is used instead of file().
  • gem install puppet-lint-fileserver-check

global_resource

  • Ensure that your manifests have no global resources.
  • gem install puppet-lint-global_resource-check

leading_zero

  • Check for unquoted numbers with leading zero.
  • gem install puppet-lint-leading_zero-check
  • gem 'puppet-lint-leading_zero-check', :require =>; false

newmericvariable

  • Extends puppet-lint to ensure that your variables are not numeric.
  • gem install puppet-lint-numericvariable

package_ensure

  • Check the ensure attribute on package resources.
  • gem install puppet-lint-package_ensure-check
  • gem 'puppet-lint-package_ensure-check'

param_docs

  • Check that validates all parameters are documented.
  • gem install puppet-lint-param-docs

resource_outside_class

  • Check if resources exist outside of a class or defined type.
  • gem install puppet-lint-resource_outside_class-check

resource_reference_syntax

  • Ensure that the reference syntax follows Puppet 4 style.
  • gem install puppet-lint-resource_reference_syntax
  • gem 'puppet-lint-resource_reference_syntax'

roles_and_profiles

  • Check that a node definition declares only a role, a role class does not have any param and only declares profiles, and a profiles class can declare anything but a role. gem install puppet-lint-roles_and_profiles-check
  • gem install puppet-lint-roles_and_profiles-check
  • gem 'puppet-lint-roles_and_profiles-check'

security

  • Checks puppet manifests for security related problems.
  • gem install puppet-lint-security-plugins

spaceship_operator_without_tag

  • Check that spaceship operator is called with a tag.
  • gem install puppet-lint-spaceship_operator_without_tag-check
  • gem 'puppet-lint-spaceship_operator_without_tag-check', :require => false

strict_indent

  • Ensure that your manifests follow a strict indentation pattern.
  • gem install puppet-lint-strict_indent-check

trailing_comma

  • Check for missing trailing commas.
  • gem install puppet-lint-trailing_comma-check
  • gem 'puppet-lint-trailing_comma-check', :require => false

trailing_newline

  • Ensure that your manifest files end with newlines.
  • gem install puppet-lint-trailing_newline-check

undef_in_function

  • Check for undef in function calls.
  • gem install puppet-lint-undef_in_function-check

unquoted_string

  • Check that selectors and case statements cases are quoted.
  • gem install puppet-lint-unquoted_string-check
  • gem 'puppet-lint-unquoted_string-check', :require => false

usascii_format

  • Check that manifest files contain only US ASCII.
  • gem install puppet-lint-usascii_format-check

variable_contains_upcase

  • Ensure that your variables are all lower case.
  • gem install puppet-lint-variable_contains_upcase

version_comparison

  • Check for versions compared as numbers.
  • gem install puppet-lint-version_comparison-check
  • gem 'puppet-lint-version_comparison-check', :require => false

vim_modeline

  • Check for vim comment (modeline) as the last line in a manifest.
  • gem install puppet-lint-vim_modeline-check

Setting up Jenkins-CI with the Docker-Plugin

This post describes how to:

What you’ll need

  • Two hosts running Linux. One for running Jenkins-CI the other for running Docker. I’ll be using Ubuntu for this how to but any distribution should do as long as it meets Jenkins and Docker’s requirements.
  • Internet access. Ubuntu and Docker both need Internet access.

While you can do everything this how to describes on one host, by using two hosts you will get a feel for the master-slave configuration we are creating.

Security Note

This how-tos’ servers are on a secure network, so the hosts’ firewalls are disabled to speed things up. If you are using CentOS, you will want to disable SELinux. In a production environment be sure to verify your environments security requirements before disabling any of these features.

On Ubuntu:

# sudo stop ufw
# sudo ufw disable

On CentOS:

# service iptables stop
# chkconfig iptales off
# setenforce 0
# sed -i 's/=enforcing/=disabled/g' /etc/sysconfig/selinux

Setting up the Docker host

  • Install the OS on what will be the Docker server. Make sure OpenSSH is installed, configured and running.
  • Install Docker by following the instructions in the documentation, http://docs.docker.com/linux/step_one/.
  • Once Docker is running properly, update the /etc/default/docker (CentOS: /etc/sysconfig/docker) file to include the following:

DOCKER_OPTS="-H unix:///var/run/docker.sock -H tcp://0.0.0.0:2375"

  • The -H unix:///var/run/docker.sock configures a local unix socket for Docker to use. This is the default.
  • The -H tcp://0.0.0.0:2375 option configures Docker to listen on a TCP port, a requirement for Jenkins to talk to Docker.

Restart Docker to enable the changes.

sudo restart docker

Optional:  if you feel confident, add your user to the docker group so you will not have to prefix your docker command with sudo all the time.

sudo usermod -aG docker myuser

You need to logout and login for the change to take effect.

Docker images

Our example uses a Docker image created by the author of the Jenkins Docker Plugin. Download the evarga/jenkins-slave image using the command:

sudo docker pull evarga/jenkins-slave

The time required to download the image depends on your Internet connection speed.

The evarga/jenkins-slave image is a Debian image with a pre-configured jenkins user account.

Setting up the Jenkins server

  • Install the OS on what will be the Jenkins-CI server. Make sure OpenSSH is installed, configured and running.
  • Go to jenkins-ci.org and download the latest Jenkins-CI package. This how-to uses the current release, but the stable version should work.

Sample Jenkins installation process for Ubuntu:

wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
echo 'deb http://pkg.jenkins-ci.org/debian binary/' > /etc/apt/sources.list.d/jenkins.list
sudo apt-get update
sudo apt-get install jenkins
sudo service jenkins start

This will:
– Install the Jenkins-CI package key.
– Configure the Jenkins-CI package repository.
– Install the Jenkins daemon and its dependencies.
– Start the Jenkins service.

Get the IP address (or hostname if DNS is configured) of your Jenkins host and using a browser connect to the Jenkins-CI host on port 8080. Example: http://192.168.0.10:8080/ If everything worked, you’ll be greeted with a Welcome to Jenkins! page.

Note:
Jenkins can take some time to start, so don't be surprised if you cannot connect right away.

By default, Jenkins-CI gives full access to the user. No user name or password is required to login. If you decide to roll-out your own setup in production, be sure to look into the various authentication and authorization options.

Installing and configuring the Jenkins Docker plugin

Installing the Docker plugin

  • Using your browser, connect to your Jenkins server.
  • From the main menu, select Manage Jenkins then select Manage Plugins.
  • If there are any updates available, install them before continuing.
  • Click on the Available tab.
  • In the Filter field at the top of the page, type in Docker. The list of
    matching plugins will update automatically.
  • Select Docker plugin from the list.
  • Click the Download now and install after restart button at the bottom of
    the page.
  • You must now restart Jenkins manually (service jenkins restart) or check off
    the Restart Jenkins when installation is complete and no jobs are running box
    which will restart Jenkins-CI for you.

Configuring the Docker plugin

Configuring the Docker plugin is a two-step process. The first step is configuring the plugin to use our Docker server. The second step is configuring the list of Docker images available to Jenkins-CI on that server.

Adding the Docker server (aka, the cloud)

I’m only describing the fields required to configure the plugin. Each field’s help is accessed by clicking the field’s help icon (?).

  • From the Manage Jenkins menu, select Configure System.
  • Scroll down to the Cloud section. (Usually at the bottom on the page.)
  • Click the Add a new cloud drop down and select Docker. A Docker sub-section is added to the Cloud section.
  • Now fill in the Docker cloud section.
  • Name: Enter a name for the docker cloud. This is purely a label and can be anything.
  • Docker URL: Enter the URL of our Docker server including the port number. This is the IP or hostname of the Docker server we created earlier. Example: http://172.16.210.34:2375
  • Credentials: We didn’t configure our Docker server to use credentials so leave it set to none.
  • Container Cap: The number of Docker containers we can run at any one time on the Docker server.
  • Click the Test Connection to test the configuration. The Docker version running on the server is returned if everything is correct.

The server is now configured and we can move on to configuring the images to use.

Configuring the Docker images to use

I’m only describing the fields required. Each field’s help can be accessed by clicking the field’s help icon (?).

  • In Cloud’s Docker sub-section, click the Add Docker Template drop down and select Docker Template. An empty Docker Template gets added to the section.
  • Fill in the Docker Template.
  • Docker Image: This is the name of the docker image to use. Our example uses the image we downloaded earlier,
    evarga/jenkins-slave.
  • Remote Filing System Root: This is the path to the Jenkins home directory inside the Docker image. If you use another Docker image, you probably need to adjust this value.
  • Labels: This is the label to assign to this image. This is the label that appears in other parts of Jenkins, including restrictions. I suggest a label based on the image name. Our example uses evarga-jenkins-slave.
  • Launch method: This is how Jenkins will log in to the running Docker image. By default there is only the SSH option which is what we’ll use.
  • Next to the empty Credentials drop down, click the Add button to open the Add credentials dialog. The username and password are already configured in the Docker image and have been provided by the image’s author.
  • Kind: Username with password
  • Username: jenkins
  • Password: jenkins
  • Description: Enter a description of the credentials. For example: evarga/jenkins-slave ssh credentials.
  • Click Add and the credentials will be added to Jenkins. Because this is the first set of credentials we created they will automatically be selected.
  • Click the Save button at the bottom of the page.

That’s it. The Docker plugin is now ready for use.

A quick example

Here is a quick example of how to use the docker plugin in a job.

  • Click New Item.
  • Enter a name for the job.
  • Select the Freestyle project option.
  • Click OK. You’ll be taken to the project’s configuration page.
  • Enable the Docker Container option.
  • Make sure Restrict where this project can be run is selected.
  • In the Label Expression field, enter evarga-jenkins-slave. Jenkins attempts to auto-complete the label from the list of labels previously defined. Jenkins will display an error if it cannot find the label you enter.
  • Click Add build step and select Execute shell.
  • We’ll use a simple script. Copy and paste the code into the Command area.
#!/bin/bash
env
echo "hello from docker!"
exit $?
  • Click Save.
  • Click the Build Now link.

Unlike a locally run job, the Docker slave jobs can take a minute or two to
start up.

If everything worked, the output from the job should look like the following:

Started by user anonymous
Building remotely on our-docker-cloud-451be36e99da (evarga-jenkins-slave) in workspace /home/jenkins/workspace/testjob
[testjob] $ /bin/bash /tmp/hudson4153896756936909309.sh
BUILD_URL=http://172.16.210.43:8080/job/testjob/1/
HUDSON_SERVER_COOKIE=5b6b248fca7784cf
SHELL=/bin/bash
SSH_CLIENT=172.16.210.43 60438 22
DOCKER_HOST=tcp://172.16.210.34:2375
BUILD_TAG=jenkins-testjob-1
WORKSPACE=/home/jenkins/workspace/testjob
JOB_URL=http://172.16.210.43:8080/job/testjob/
USER=jenkins
JENKINS_HOME=/var/lib/jenkins
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
MAIL=/var/mail/jenkins
_=/usr/bin/env
PWD=/home/jenkins/workspace/testjob
HUDSON_URL=http://172.16.210.43:8080/
JOB_NAME=testjob
BUILD_DISPLAY_NAME=#1
JENKINS_URL=http://172.16.210.43:8080/
BUILD_ID=1
DOCKER_CONTAINER_ID=451be36e99dabf23221bfc627462d5d61b8c24971b74a814e04aa270d66fef2b
HOME=/home/jenkins
SHLVL=2
JENKINS_SERVER_COOKIE=5b6b248fca7784cf
EXECUTOR_NUMBER=0
NODE_LABELS=evarga-jenkins-slave our-docker-cloud-451be36e99da
LOGNAME=jenkins
JENKINS_CLOUD_ID=our-docker-cloud
HUDSON_HOME=/var/lib/jenkins
SSH_CONNECTION=172.16.210.43 60438 172.31.31.99 22
NODE_NAME=our-docker-cloud-451be36e99da
BUILD_NUMBER=1
HUDSON_COOKIE=285ada53-ada0-48d1-9766-f806a2e45e7f
hello from docker!
Finished: SUCCESS

Congratulations! You know have a working Jenkins-CI setup using Docker for SSH build slaves.