This question comes up a lot when I’m talking about Linux Provisioning with people, and came up during a recent online conference event, so I decided to feed the commentary to Google and Fedora Planet. This is just my opinion, not my employer’s or anything like that, and there are always exceptions to the rule.
I’ll start with the conclusion: IMHO, for most Linux datacenter setups, kickstart is better for most people. I will explain why.
Typically folks will want to “do images” for several reasons. One is the perception that it is “faster”. This is a misleading thing, for several reasons. On most networks, a fully automated kickstart install will take somewhere in the range of 4-8 minutes, whether physical or virtual. While it seems that simply copying an image is faster, the problem is in the time the admin takes to /prepare/ the image, which can be considerably longer. When deciding whether you’d want to waste valuable human time or machine time, I’d rather waste that tiny bit of machine time. In rare cases, you might want to reinstall an entire cluster of 600 machines every night — but how often does that come up?
Also, on a large network, the cost of transferring images to clone physical systems may not me much better than the kickstart install speed. Depending on your setup, this can vary. They may be faster “right now” but maintenance is not fast.
People frequently call images “golden images”. I believe this is because the word golden makes them sound like they are made of gold and therefore better. But… there are numerous problems with this. There’s no good way to know exactly /what/ goes into an image. For this reason, they really should be called “pyrite images”. I don’t pretend to be an expert on SARBOX compliance and such, but it’s very important that you know exactly what goes into everything — I know what it feels like to inherit some bits at a new job and have no idea how to recreate it from scratch. That’s bad. So unless your image is generated programmatically, that’s a bad thing. Always using fully automated kickstarts and config management tools will make you sure that you /can/ do that. While on the other hand if you are using kickstart (say with Cobbler) and a config management system (like Puppet) you have a text description of exactly what you told all of your systems to do, that you can generate again at will. At this point if you are running systems management tools against an image to reconfigure it, or cracking it open, etc, at this point you are not saving terribly much time. You are probably losing your time, and losing more of your time than you are losing of machine time. Note: if you have a minion that makes these edits for you, you are lucky, and none of this applies. However your minion would probably like to spend less time on changes too.
Images may not work very well for different kinds of hardware. Certain different systems may require different kernel arguments, or may have different storage configurations. They may have different video setups. Where one kickstart for a base install will work to perform a task in the same way across a wide variety of systems, images would have to be created for non-homogenous systems.
Finally, there are also various things with images that you have to fix after you deploy them. Networking configurations are certaintly one of them — particulary in a static setup, this can be quite fun. Further, you will need to make sure MAC addresses, SSH keys, and certifcates from tools like Puppet and Func are not duplicated. So there is extra overhead you have to do to “fix” various images after they are deployed.
This starts to feel really hacked up after a while, and it’s not really easy to declare all of the things that you might break.
Kickstarts, on the other hand, offer lots of nice advantages. It’s easy to compare two different install setups in a kickstart because they are just text. You can easily diff one between another, or add a new package. Surgery on a install definition that is done via a kickstart and config management tool can happen very very quickly (adding a package for instance, is a one line insertion in a text file), where as with an image you have to fire up the machine, perform a step, shut it down, and copy it.
Kickstart also works with cool tools like livecd-creator, so you can make live versions of your setups.
The only penalty is the machines take just a little bit longer to deploy, but you win a really good audit trail, better updates, and the advantage of all the work that has gone into the installers to make them work with varying hardware. Especially with a system like Cobbler, you can also template out differences on a system by system basis really easily — such as setting up different networking parameters for different systems or different services and so on — all at install time — which is not easily done with an image based system. This kind of system-by-system templating by passing variables is very powerful, and you don’t get that with images. You also don’t get “snippets” with images, so if you want to do the same thing to multiple images, you have to have tooling around that or you will probably not be too happy. It’s ok if you have maybe one image, but if you have more? Ow.
Images are also pretty bad at updates. If you want to update an image, you have to start it up and update it — or rebuild a new one. While with a live setup process you can make sure you have the latest of everything all the time. If you’re just going to deploy an image and then run “yum update” on it, you really haven’t saved any time — because you could have installed those updated packages at install time.
I won’t deny that there are reasons where images may be better for particular uses — Windows desktops are likely homogenous, for instance, and maybe that is useful there — but I don’t think they are the best option on Linux. You might also be deploying the same image across a wide variety of machines and hosting things primarily in VMs. In this case, maybe the provisioning time savings is important — though in that case, at least make sure that you have a consistent plan for how you are going to rebuild them, and make sure that process is automated and scripted, and does not involve a lot of manual human effort.
If you’re worried about the 4-8 minute install time lag and so forth, I think its’ a very good reason to invoke XKCD #303, knowing full well the audit trail and added flexibility will be useful later, and making changes to the “golden” images aren’t ever going to make you stay late on a Friday, and you know exactly what is going to each box that you have.
Anyway, my two cents on why kickstart still has a good reason to stick around, and how in many ways, waiting just a few more minutes for install can make your life in managing those installs a lot more fun, a lot simpler, and a lot more flexible.
Thanks for sharing your opinions. I would very much like to hear your opinion as well, not on deploying data-centers, but rather on say hundreds of user desktops either sharing one golden image (say ro NFS root), vs managing the configs manually/puppet, vs maybe running sessions on servers and thing clients
Hi Ahmed,
I’d personally kickstart those as well and manage them in the same way (probably running config management manually rather than checking in periodically, in the case of laptops). Hundreds is not that many, and the same reasons about being able to make changes apply. You could just use straight kickstart. Maintaining the images adds to management overhead, and the Stateless project ideas that are /probably/ needed for this sort of thing really aren’t actively evolving: https://www.redhat.com/archives/stateless-list/2009-June/msg00001.html . I think most people have realized the extra complexity really isn’t needed. In the case of schools, reinstalling machines on a regular basis is fairly standard for Windows systems, but if we’re talking Linux, simply offering it up as a PXE menu option should be sufficient. Then you can just put home dirs on the network or whatever you want.
I don’t have much thin client experience… there is K12 LTSP and such, though I have never really encountered thin clients in the wild so it seems that solution is not that commonly used. At this point computers capable of running a desktop are pretty cheap so I think that has a lot of bearing on it.
Mike,
That is great
I’ve always found using images in the datacenter to be a self correcting problem. Eventually, one day, the guy doing the images will discover kickstart and the problem solves itself
I really liked this overview on your opinions of the two options, especially since I feel exactly the same way! Having come from an IT environment where imaging Windows systems was frequent, as well as a college program that touched on unattended Windows installation, kickstarts are a revelation! They are so much more powerful, and installation of a kickstarted Fedora install is faster than any Windows installation I’ve ever dealt with.
I’m so glad I wasn’t the guy who had to create the images where I worked, but I did discuss the process with him. What a mess! He had to create at least one image per hardware configuration, of which there were at least 3 at a given time. Not to mention, we upgraded about 1/3 of the computers a year!
On top of that, another thing that kickstarts can do that imaging cannot is automated software installation and configuration for any programs you need. There were some Windows programs that, for whatever reason, could not be part of the image, so we had another guy whose job was to package those in MSI format so they could later be installed after the imaging process was done. So slow and painful, and the work could have been done by one part-time employee with kickstarts!
Greetings,
I don’t manage “data-center” machines but rather lab machines in a college environment. All of the computers in each lab are all the same.
Making one “golden image” does take time… it takes just as much time as customising a single machine post kickstart install. In our labs we do have a considerable amount of third party software that has to be added (BlueJ, Greenfoot, Matlab, etc) that would take a bit of effort to automate via kickstart post-install scripting… rsync perhaps.
I’m not trying to say that kickstart isn’t better in some scenarios but for me, where I have triple-boot machines (PCs – Windows XP, CentOS 5.3, and Fedora 11 ; iMacs – OS X, Windows XP, Fedora 11).
I use a few external USB drives and partimage. For Mac OS X I use their Disk Utility app to create an image and to restore it.
Kickstart can only do Linux whereas imaging lets me do multiple ones. On machines with just Linux and not much in the way of third-party apps, I would definitely consider kickstart.
While you say it takes just as long to customize a kickstart for the first time, once customized, the point is tweaking it is much easier and can be done in seconds, while with an image based setup you repeat that pain again and again.
Of course, your point about triple booting and such is valid, though in those cases, you might could also consider running the Windows machines as guests inside Linux — making them the odd thing out.
This doesn’t really solve any problems with your iMacs though — because of their hardware locking. Unfortunate, I actually do like OS X… but mostly just for a desktop, it is not so friendly as a development environment.
Though for machines that didn’t have to run on OS X, running Windows as a virt guest might make things easier. I would rather prefer to pretend it didn’t exist
I guess the real question is why you need the Windows machines and the Macs
The college setups I’ve seen didn’t dual boot — Linux machines pretty much ran Linux. Life would definitely get better if you eliminated that, then you don’t have the fun of maintaining the uber-triple image.
If you have the Macs and have to use them, though I’m not happy with it entirely, I’d almost be tempted to get Virtual Box or something working there and use them as hosts. The sucky part there is you have an extra layer in front of using the actual terminal and such.
If I have a Mac setup, I’d almost rather be SSH’ed into a VM rather than having to use those consoles, though you could do that in virt too.