How and Why?
Let’s say you have a server that has been lovingly hand-crafted that you want to containerize.
Figuring out exactly what software is required on there and what config files need adjustment would be quite a task, but fortunately blueprint exists as a solution to that.
What I’ve done here is automate that process down to a few simple steps. Here’s how it works:
You kick off a ShutIt script (as root) that automates the bash interactions required to get a blueprint copy of your server, then this in turn kicks off another ShutIt script which creates a Docker container that provisions the container with the right stuff, then commits it. Got it? Don’t worry, it’s automated and only a few lines of bash.
There are therefore 3 main steps to getting into your container:
– Install ShutIt on the server
– Run the ‘copyserver’ ShutIt script
– Run your copyserver Docker image as a container
Step 1
Install ShutIt as root:
sudo su -
pip install shutit
The pre-requisites are python-pip, git and docker. The exact names of these in your package manager may vary slightly (eg docker-io or docker.io) depending on your distro.
You may need to make sure the docker server is running too, eg with ‘systemctl start docker’ or ‘service docker start’.
Step 2
Check out the copyserver script:
git clone https://github.com/ianmiell/shutit_copyserver.git
If you like this, you might like one of my books:
Learn Bash the Hard Way
Learn Git the Hard Way
Learn Terraform the Hard Way
Step 3
Run the copy_server script:
cd shutit_copyserver/bin ./copy_server.sh
There is a prompt to ask what docker base image you want to use. Make sure you use one as close to the original server as possible.
Step 4
Run the built image:
docker run -ti [username]/copyserver /bin/bash
You are now in a practical facsimile of your server within a docker container!
zwischenzugs 
I am using amazon linux and I got this error:
Pause point on fail: Could not determine Linux distro information. Please inform ShutIt maintainers.
Hi Aly, Can you raise a ticket in GitHub and we can pursue there? What kind of distro is ‘Amazon Linux’ under the hood?
https://aws.amazon.com/amazon-linux-ami/2016.03-release-notes/
which linux distro/version are tested and confirmed to work? -thanks.
I tried it with centos 7 but got this error:
[root@ip-10-66-118-190 centos]# cd shutit_copyserver/bin
[root@ip-10-66-118-190 bin]# ./copy_server.sh
/home/centos/shutit_copyserver /home/centos/shutit_copyserver/bin
# ShutIt Started…
# Loading configs…
2016-08-13 14:34:12,551 INFO: ShutIt modules loaded
2016-08-13 14:34:12,552 INFO: Configuration loaded
2016-08-13 14:34:12,618 INFO: Connected to target
Please input a value for shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver.base_image
Please input an appropriate docker base image, eg:
ubuntu
ubuntu:12.04
centos
You can use any base image you want, but be aware that non-apt or yum-based images are less likely to succeed.:
centos
2016-08-13 14:34:15,007 INFO: Considering whether to build: shutit.tk.setup
2016-08-13 14:34:15,087 INFO: Building ShutIt module: shutit.tk.setup with run order: 0
2016-08-13 14:34:15,090 INFO: Considering whether to build: shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver
2016-08-13 14:34:15,177 INFO: Building ShutIt module: shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver with run order: 1161086295
2016-08-13 14:34:15,177 INFO: Matching output from: “whoami” to one of these regexps:root
2016-08-13 14:34:15,179 INFO: Installing package: python-pip
2016-08-13 14:34:15,181 INFO: Sending: yum install -y python-pip
2016-08-13 14:34:18,776 INFO: Output (squashed): Loaded plugins: fastestmirrorL […]
2016-08-13 14:34:18,777 INFO: Installing package: git
2016-08-13 14:34:18,779 INFO: Sending: yum install -y git
2016-08-13 14:34:22,255 INFO: Output (squashed): Loaded plugins: fastestmirrorL […]
2016-08-13 14:34:22,256 INFO: Matching output from: “git config -l” to one of these regexps:user.email
2016-08-13 14:34:22,259 INFO: Matching output from: “git config -l” to one of these regexps:user.name
2016-08-13 14:34:22,261 INFO: Sending: rm -rf /tmp/shutit_copyserver
2016-08-13 14:34:22,263 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 14:34:22,264 INFO: Sending: pip install blueprint
2016-08-13 14:34:22,539 INFO: Output (squashed): Requirement already satisfied […]
2016-08-13 14:34:22,540 INFO: Sending: echo ‘:package:apt/docker’ >> /etc/blueprintignore
2016-08-13 14:34:22,542 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 14:34:22,543 INFO: Sending: echo ‘:package:apt/docker.io’ >> /etc/blueprintignore
2016-08-13 14:34:22,544 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 14:34:22,545 INFO: Sending: echo ‘:package:apt/docker-engine’ >> /etc/blueprintignore
2016-08-13 14:34:22,547 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 14:34:22,548 INFO: Sending: echo ‘:package:apt/virtualbox’ >> /etc/blueprintignore
2016-08-13 14:34:22,550 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 14:34:22,551 INFO: Sending: echo ‘:package:apt/virtualbox-dkms’ >> /etc/blueprintignore
2016-08-13 14:34:22,553 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 14:34:22,554 INFO: Sending: echo ‘:package:apt/virtualbox-qt’ >> /etc/blueprintignore
2016-08-13 14:34:22,556 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
Traceback (most recent call last):
File “/bin/shutit”, line 11, in
sys.exit(main())
File “/usr/lib/python2.7/site-packages/shutit_main.py”, line 609, in main
do_build()
File “/usr/lib/python2.7/site-packages/shutit_main.py”, line 421, in do_build
build_module(module)
File “/usr/lib/python2.7/site-packages/shutit_main.py”, line 362, in build_module
if not module.build(shutit):
File “/usr/lib/python2.7/site-packages/shutit_module.py”, line 58, in wrapper
ret = func(self, shutit)
File “/home/centos/shutit_copyserver/shutit_copyserver.py”, line 80, in build
shutit.send(shutit.cfg[‘host’][‘shutit_path’] + ‘/shutit skeleton –output_dir –base_image ‘ + shutit.cfg[self.module_id][‘base_image’] + ‘ –template_branch docker –module_directory /tmp/shutit_copyserver –module_name copied_server –domain shutit.copied_server –delivery docker’)
KeyError: ‘host’
/home/centos/shutit_copyserver/bin
[root@ip-10-66-118-190 bin]#
After a busy few months I have some time to look at this. There are a couple of bugs I’m fixing, and testing now.
Thanks for this – I’ve pushed a change which should help.
I pulled the update and retried again, but got this new error, please check:
[root@ip-10-66-115-25 centos]# cd shutit_copyserver/bin
[root@ip-10-66-115-25 bin]# ./copy_server.sh
/home/centos/shutit_copyserver /home/centos/shutit_copyserver/bin
# ShutIt Started…
# Loading configs…
2016-08-13 16:07:36,712 INFO: Correcting insecure file permissions on: configs/build.cnf
2016-08-13 16:07:36,721 INFO: ShutIt modules loaded
2016-08-13 16:07:36,722 INFO: Configuration loaded
2016-08-13 16:07:36,787 INFO: Connected to target
Please input a value for shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver.base_image
Please input an appropriate docker base image, eg:
ubuntu
ubuntu:12.04
centos
You can use any base image you want, but be aware that non-apt or yum-based images are less likely to succeed.:
centos
2016-08-13 16:07:39,115 INFO: Considering whether to build: shutit.tk.setup
2016-08-13 16:07:39,196 INFO: Building ShutIt module: shutit.tk.setup with run order: 0
2016-08-13 16:07:39,198 INFO: Considering whether to build: shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver
2016-08-13 16:07:39,284 INFO: Building ShutIt module: shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver with run order: 1161086295
2016-08-13 16:07:39,284 INFO: Matching output from: “whoami” to one of these regexps:root
2016-08-13 16:07:39,288 INFO: Installing package: python-pip
2016-08-13 16:07:39,289 INFO: Sending: yum install -y python-pip
2016-08-13 16:07:42,803 INFO: Output (squashed): Loaded plugins: fastestmirrorL […]
2016-08-13 16:07:42,804 INFO: Installing package: git
2016-08-13 16:07:42,806 INFO: Sending: yum install -y git
2016-08-13 16:07:46,258 INFO: Output (squashed): Loaded plugins: fastestmirrorL […]
2016-08-13 16:07:46,259 INFO: Matching output from: “git config -l” to one of these regexps:user.email
2016-08-13 16:07:46,261 INFO: Sending: git config –global user.email “shutit@shutit.tk”
2016-08-13 16:07:46,264 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:46,265 INFO: Matching output from: “git config -l” to one of these regexps:user.name
2016-08-13 16:07:46,267 INFO: Sending: git config –global user.name “ShutIt”
2016-08-13 16:07:46,270 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:46,271 INFO: Sending: rm -rf /tmp/shutit_copyserver
2016-08-13 16:07:46,273 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:46,274 INFO: Sending: pip install blueprint
2016-08-13 16:07:47,045 INFO: Output (squashed): Collecting blueprint Download […]
2016-08-13 16:07:47,046 INFO: Sending: echo ‘:package:apt/docker’ >> /etc/blueprintignore
2016-08-13 16:07:47,048 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:47,049 INFO: Sending: echo ‘:package:apt/docker.io’ >> /etc/blueprintignore
2016-08-13 16:07:47,050 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:47,051 INFO: Sending: echo ‘:package:apt/docker-engine’ >> /etc/blueprintignore
2016-08-13 16:07:47,053 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:47,054 INFO: Sending: echo ‘:package:apt/virtualbox’ >> /etc/blueprintignore
2016-08-13 16:07:47,056 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:47,057 INFO: Sending: echo ‘:package:apt/virtualbox-dkms’ >> /etc/blueprintignore
2016-08-13 16:07:47,059 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:47,060 INFO: Sending: echo ‘:package:apt/virtualbox-qt’ >> /etc/blueprintignore
2016-08-13 16:07:47,062 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:47,063 INFO: Sending: shutit skeleton –output_dir –base_image centos –pattern docker –name /tmp/shutit_copyserver –domain shutit.copied_server –delivery docker
2016-08-13 16:07:47,263 INFO: Output (squashed): # ShutIt Started… # […]
2016-08-13 16:07:47,264 INFO: Sending: cd /tmp/shutit_copyserver
2016-08-13 16:07:47,266 INFO: Output (squashed): shutit.shutit_copyserver.shuti […]
2016-08-13 16:07:47,267 INFO: Sending: blueprint create shutit_copyserver
2016-08-13 16:08:06,971 INFO: Output (squashed): # [blueprint] searching for AP […]
2016-08-13 16:08:06,974 INFO: Fixing up your terminal, please wait…
================================================================================
WARNING: command:
blueprint create shutit_copyserver
returned unaccepted exit code: 1
If this is expected, pass in check_exit=False or an exit_values array into the send function call.
Interactive, so not retrying.
Pause point on exit_code != 0 (1). CTRL-C to quit
================================================================================
You now have a standard shell. Hit CTRL and then ] at the same time to continue ShutIt run.
/tmp/shutit_fixterm
shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver#cHAQmuDJ>
OK,so this time can you post the output of the same command with -l debug (btw this is probably better pursued in a github issue)!
Thanks a lot for the help, I created a new git issue and added the debug output here:
https://github.com/ianmiell/shutit_copyserver/issues/3
Hi! I have tried to create the copy of my server (Ubuntu 16.04.2) but the script fails with the following output. Can you please help me?
root@sputnik:~/shutit_copyserver/bin# ./copy_server.sh
~/shutit_copyserver ~/shutit_copyserver/bin
# ShutIt Started…
# Loading configs…
2017-05-03 16:38:59,270 INFO: Correcting insecure file permissions on: configs/build.cnf
2017-05-03 16:38:59,282 INFO: ShutIt modules loaded
2017-05-03 16:38:59,283 INFO: Configuration loaded
2017-05-03 16:39:01,530 INFO: Connected to target
Please input a value for shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver.base_image
Please input an appropriate docker base image, eg:
ubuntu
ubuntu:12.04
centos
You can use any base image you want, but be aware that non-apt or yum-based images are less likely to succeed.:
ubuntu
2017-05-03 16:39:16,880 INFO: Considering whether to build: shutit.tk.setup
2017-05-03 16:39:18,218 INFO: Building ShutIt module: shutit.tk.setup with run order: 0
2017-05-03 16:39:18,372 INFO: Considering whether to build: shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver
2017-05-03 16:39:19,740 INFO: Building ShutIt module: shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver with run order: 1161086295
2017-05-03 16:39:19,740 INFO: Matching output from: “whoami” to one of these regexps:root
2017-05-03 16:39:19,793 INFO: Installing package: python-pip
2017-05-03 16:39:19,896 INFO: python-pip already installed.
2017-05-03 16:39:19,896 INFO: Installing package: git
2017-05-03 16:39:19,995 INFO: git already installed.
2017-05-03 16:39:19,996 INFO: Matching output from: “git config -l” to one of these regexps:user.email
2017-05-03 16:39:20,049 INFO: Matching output from: “git config -l” to one of these regexps:user.name
2017-05-03 16:39:20,103 INFO: Sending: rm -rf /tmp/shutit_copyserver
2017-05-03 16:39:20,186 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:20,238 INFO: Sending: pip install blueprint
2017-05-03 16:39:20,784 INFO: Output (squashed): Requirement already satisfied: blueprint in /usr/local/lib/python2.7/dist-packagessputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K# […]
2017-05-03 16:39:20,836 INFO: Sending: echo ‘:package:apt/docker’ >> /etc/blueprintignore
2017-05-03 16:39:20,890 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:20,942 INFO: Sending: echo ‘:package:apt/docker.io’ >> /etc/blueprintignore
2017-05-03 16:39:20,996 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:21,048 INFO: Sending: echo ‘:package:apt/docker-engine’ >> /etc/blueprintignore
2017-05-03 16:39:21,103 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:21,155 INFO: Sending: echo ‘:package:apt/virtualbox’ >> /etc/blueprintignore
2017-05-03 16:39:21,209 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:21,261 INFO: Sending: echo ‘:package:apt/virtualbox-dkms’ >> /etc/blueprintignore
2017-05-03 16:39:21,316 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:21,368 INFO: Sending: echo ‘:package:apt/virtualbox-qt’ >> /etc/blueprintignore
2017-05-03 16:39:21,422 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:21,474 INFO: Sending: shutit skeleton –output_dir –base_image ubuntu –pattern docker –name /tmp/shutit_copyserver –domain shutit.copied_server –delivery docker
2017-05-03 16:39:21,768 INFO: Output (squashed): # ShutIt Started… # Loading configs…# Run:cd /tmp/shutit_copyserver && ./build.sh# to build.# And then:./run.sh# to run.sputnik:shutit.shutit_copyserver.shu […]
2017-05-03 16:39:21,820 INFO: Sending: cd /tmp/shutit_copyserver
2017-05-03 16:39:21,872 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:39:21,924 INFO: Sending: blueprint create shutit_copyserver
2017-05-03 16:41:17,936 INFO: Output (squashed): # [blueprint] parsing blueprintignore(5) rulesdpkg-query: Kein Paket gefunden, das auf python3:any passtdpkg-query: Kein Paket gefunden, das auf docker-engine p […]
2017-05-03 16:41:17,987 INFO: Sending: blueprint show -S shutit_copyserver
2017-05-03 16:41:19,067 INFO: Output (squashed): shutit_copyserver/bootstrap.shsputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:41:19,118 INFO: Sending: sed -i ‘s/^…n “$SERVICE_.*$//g’ shutit_copyserver/bootstrap.sh
2017-05-03 16:41:19,212 INFO: Output (squashed): sputnik:shutit.shutit_copyserver.shutit_copyserver.shutit_copyserver:Koy9EA0K#
2017-05-03 16:41:26,349 INFO: ShutIt run finished
After a busy few months I have some time to look at this. There are a couple of bugs I’m fixing, and testing now.
Just tried to run through this on one of our ec2 instances and I ran into the following problem. Any ideas?
Thanks in advance!
[root@ip-172-31-8-239 bin]# ./copy_server.sh
~/shutit_copyserver ~/shutit_copyserver/bin
/usr/lib/python2.6/dist-packages/shutit_pexpect.py:46: DeprecationWarning: the md5 module is deprecated; use hashlib instead
from md5 import md5
Traceback (most recent call last):
File “/usr/bin/shutit”, line 11, in
sys.exit(main())
File “/usr/lib/python2.6/dist-packages/shutit.py”, line 50, in main
if sys.version_info.major == 2:
AttributeError: ‘tuple’ object has no attribute ‘major’
~/shutit_copyserver/bin
After a busy few months I have some time to look at this. At first glance though, shutit only supports 2.7+.
Hi zwischenzugs,
I am trying to run ./copy_server.sh. But getting the below error. I am running it in an amazonlinux system (yum). Is there an issue due to this OS? Please help.
ShutIt version: 1.0.27
Linux ip-10-0-2-39 3.14.42-31.38.amzn1.x86_64 #1 SMP Wed May 13 20:33:05 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
2017-10-04 11:50:18,809 INFO: Installing package: python-pip
Traceback (most recent call last):
File “/usr/local/bin/shutit”, line 11, in
sys.exit(main())
File “/usr/local/lib/python2.7/site-packages/shutit.py”, line 67, in main
shutit.setup_shutit_obj()
File “/usr/local/lib/python2.7/site-packages/shutit_class.py”, line 3942, in setup_shutit_obj
self.do_build()
File “/usr/local/lib/python2.7/site-packages/shutit_class.py”, line 4181, in do_build
self.build_module(module)
File “/usr/local/lib/python2.7/site-packages/shutit_class.py”, line 4123, in build_module
if not module.build(self):
File “/usr/local/lib/python2.7/site-packages/shutit_module.py”, line 59, in wrapper
ret = func(self, shutit)
File “/opt/shutit_copyserver/shutit_copyserver.py”, line 66, in build
shutit.install(‘python-pip’)
File “/usr/local/lib/python2.7/site-packages/shutit_class.py”, line 1798, in install
loglevel=loglevel)
File “/usr/local/lib/python2.7/site-packages/shutit_pexpect.py”, line 1393, in install
package = package_map.map_packages(package, self.current_environment.install_type)
File “/usr/local/lib/python2.7/site-packages/package_map.py”, line 108, in map_packages
res = res + ‘ ‘ + map_package(package,install_type)
TypeError: cannot concatenate ‘str’ and ‘function’ objects
please try to do it again using salt stack