Automated Network Appliance Configuration Backups with Expect

An often overlooked aspect of managing network appliances like switches and routers is the necessity of configuration backups. Changes to these devices are usually infrequent, making backups easy to forget after finalizing a configuration. What’s more, network appliances often lack built-in tools for automatically copying configuration to a secure backup location. The problem becomes even more cumbersome when you consider that changes on one appliance often means changing several others. For example, if my server needs to change VLANs, a configuration change on the switch is required at minimum, but will likely be accompanied by a change to firewall rules on a router and policy changes on a security appliance. A simple VLAN reassignment is deceptively complex.

I’ve always believed backups should be automated; you simply can’t trust a user (or an administrator, for that matter) to always remember to perform them. With that in mind, I’ve implemented an automatic backup solution that overcomes these challenges by leveraging the one thing these devices have in common: a command-line interface.

Most network devices, most notably but not exclusively Cisco devices, store their configuration as a simple list of commands; if you copy the commands, you’ve created a backup. Manually copying and pasting “show run” output, while effective, is not ideal due to the forgetfulness factor. What’s the alternative then? Automatically copying and pasting output. Enter the ‘expect’ command.

I’ve had a lot of success using ‘expect’ to create backups. The ‘expect’ tool is described as “programmed dialogue with interactive programs.” In other words, we can use expect with a CLI to automate tasks that were designed to be performed interactively by a user. By leaning on ‘expect’, we gain several advantages:

  • No need to maintain a TFTP server, the traditional backup method for Cisco devices.
  • No need activate HTTP interfaces on devices solely for CURLing configurations.
  • Backups are secured over-the-wire using SSH.

Getting Started

The things you’ll need to get started are modest and you likely have them setup already:

  • A service account on each appliance accessible via SSH.
  • A Linux server that performs and/or stores backups.

If you’ve got those things, you can dive right in! The workflow for each backup will look something like this:

  1. SSH into a network appliance with a service account.
  2. Execute a “show run” or equivalent command and capture the output.
  3. After saving a copy, parse the raw output to a readily-restorable format.

Working with Expect

First, install ‘expect’ on your Linux server:

# RHEL / CentOS
yum install expect

# Debian / Ubuntu
apt-get install expect

After ‘expect’ is installed, remote into your network appliance and perform a “show run” command. Take note of all the key presses along the way, including user authentication input and returned output; these are the actions you’ll have to teach ‘expect’ to perform.

Next, create a wrapper script that will be execute an ‘expect’ script among other things. Utilizing a wrapper increases re-usability and helps protect login credentials.

#!/bin/bash

/usr/bin/expect /path/to/expect/script/cisco3560.exp "$HOST" "$USERNAME" "$PASSWORD" "$SECRET" > config_backup.raw

As you can see, we’re simply passing arguments to the ‘expect’ command and redirecting the output to a file. The first argument is the path of the expect script which describes how to interact with the device; you’ll likely need one expect script for each device type. The remaining arguments are values that will be passed to ‘cisco3560.exp’.

Our ‘cisco3560.exp’ script looks like this:

#!/usr/bin/expect
# How long this script can idle
set timeout 60
# How big the internal buffer is. Increase when dealing with lots of output.
match_max 10000

# Set variable values from command line arguments
set HOST [lindex $argv 0]
set USER [lindex $argv 1]
set PASS [lindex $argv 2]
set SECRET [lindex $argv 3]

#Start the SSH session
spawn ssh -l $USER $HOST
#Sleep 5 because sometimes the device is slow to wakeup
sleep 5
#Expect the password prompt
expect "*@*password:"
#Send the password
send "$PASS\r"
#Expect the main menu
expect "*>"
send "enable\r"
expect "Password:*"
send "$SECRET\r"
expect -re {[A-Za-z0-9\.-]+#$}
# Don't page the output, show it all at once
send "terminal length 0\r"
expect -re {[A-Za-z0-9\.-]+#$}
send "show run\r"
expect -re {[A-Za-z0-9\.-]+#$}
# reset the pager
send "terminal length 50\r"
expect -re {[A-Za-z0-9\.-]+#$}
send "exit\r"
expect eof

This particular script is about as simple as it gets but it clearly demonstrates the basic concepts. After setting some variables and spawning an SSH session, there are only two things we need to do: ‘send’ input and ‘expect’ output.

We use the ‘send’ function for inputting keystrokes including carriage returns (\r). We must be explicit; forget a carriage return and expect will hang.

We use the ‘expect’ function to tell the script what output it’s supposed to receive so it can move on. If unexpected output is returned, ‘expect’ will hang until the timeout value is reached. We can describe expected text literally or through regular expressions. We can also define procedures based on the output. Look at this notated example from the ‘expect’ man page:

expect {
  # Expect the password prompt
  Password: {
    stty -echo
    # Tell the user what's going on
    send_user "password (for $user) on $host: "
    expect_user -re "(.*)\n"
    send_user "\n"
    send "$expect_out(1,string)\r"
    stty echo
    exp_continue
  } incorrect {
    # If the device returns 'incorrect', tell the user
    send_user "invalid password or account\n"
    exit
  } timeout {
    # If the device times out, tell the user
    send_user "connection to $host timed out\n"
    exit
  } eof {
    # If the device closes the connection, tell the user.
    send_user "connection to host failed: $expect_out(buffer)"
    exit
   # Expect the prompt again, with $prompt being a regular expression.
  } -re $prompt
}

Defining procedures in this manner still only skims the surface of what ‘expect’ can do. With mastery, we can create very flexible and useful scripts that can assist us in performing tasks or completely automate interactive processes. Have a look at this article for a much more comprehensive look at ‘expect’.

Making Readily-Restorable Backups

Let’s get back to our backup. Remember that our wrapper script is capturing the raw output from the ‘expect’ command including all of the prompts and key strokes. If our goal is to create a readily-restorable backup, we need to edit the resulting file and get rid of text we don’t need. Fortunately, manipulating text is trivial. Let’s add to our wrapper script:

#!/bin/bash

/usr/bin/expect /path/to/expect/script/cisco3560.exp "$HOST" "$USERNAME" "$PASSWORD" "$SECRET" > config_backup.raw

# Delete the first 11 linues and the last 4 lines
# Put the resulting text into a separate file
sed -e '1,12d' config_backup.raw | head -n -4 > config_backup

The amount of text manipulation you need to do perform varies from device to device. A Cisco 3560 switch produces just a few unnecessary lines, and always in the same place, so I’ve chosen to simply delete those lines. Other devices might require more complex text maniuplation. I recommend always saving a copy of the raw output along with the formatted file. This ensures that, if the output changes slightly and we delete something important, we can reference the raw file that still has everything we need.

All that’s left is to incorporate this script into our backup routine. In production, my wrapper script contains a loop that will connect to every 3560 we have on the network and create separate files for each. I also have scripts to handle a pair of Cisco SGE2010s, a half-dozen Cisco 2960s, and an old TrendNET gigabit switch. If there’s a CLI interface for it, ‘expect’ can work with it!