Introducing AnsibleWorks

While it’s already over most of the internet so far (thanks GigaOM!), I haven’t posted to THIS particular blog quite yet.

Today is a great day.  I’m sitting here in lovely Montecito, CA, and have just kicked off AnsibleWorks.  I’ll be serving as CTO,with Said Ziouani, formerly of Red Hat and Eucalyptus, serving as CEO.   Great times are ahead.

You can read a lot more on blog.ansibleworks.com — but I’m hugely looking forward to taking Ansible to a whole new level and also continuing to continue [sic] all the great things that we’ve done together as a group so far.

The time has come to simplify IT to we can accelerate the pace of technological innovation — not just configuration management, but software deployment, true multi-tier server orchestration, everything.  AnsibleWorks is going to be all about making that happen.   

This is going to enable us to take Ansible forward in even more new directions, and also reach out to new audiences (you may wish to join us at AnsibleFEST in Boston this June, or stop by for a training class) — while helping enterprises take better control over their IT management.

You can follow the company on twitter at @AnsibleWorks and me at @laserllama

Warp factor nine, engage.

Preseeding Ubuntu 12.04 (for Ansible and other things)

Preseeding Debian or Ubuntu can be a difficult experience compared to kickstart, which is a generally killer way to automate a distro installation.

What follows in the gist below is an ansible tasks list (you’ll have to do some legwork on your own) that templates out a preseed file for use with cobbler, and a template for a post.sh script that will run in the post of the preseed file to set up sudoers and authorized_keys.

https://gist.github.com/4707732

The real trick is that (A) the late_command line in a preseed can only be one line, and (B) much of the magic happens in a chroot, as indicated by ‘in-target’.

Another major gotcha is that many items that appear to be setable in a preseed must still be set by kernel arguments.   The options I used, as shown from “cobbler system dumpvars” are shown in the gist.

95% of the preseed syntax deals with base OS setup, and you will most definitely want to modify things, and search for {{ foo }} (Jinja2 variable substitution) and so on to replace values that were used in my templates that may not make sense for you.   Choose a sudoers configuration that is appropriate to your environment.

The bottom line is that with sufficient legwork, yes, you can whip Ubuntu into shape, but kickstart remains a superior installation scripting solution.  

Ansible 1.0 released!

Ansible 1.0 is a major milestone in the development of Ansible.cc. 

We have added 6 great new modules:

• pacman (package management for Arch linux)
• sysctl
• apt_key for installing new apt signing keys (Debian/Ubuntu)
• ec2_facts which contact the EC2 metadata service
• hg (Mercurial)
• pkgin (package management for Joyent Smart OS)

We have also acquired a wide variety of useful new features.  Some of the highlights include:

• Support for alternative sudo implementations
• New conditionals ‘when_failed’ and ‘when_changed’
• Being able to change the connection type in the middle of a playbook
• Task includes of infinite depth
• If you want hashes to “merge” instead of override each other when names collide, you can now configure that in the configuration file.
• Hostnames in the inventory file can now be indicated by alphabetic in addition to the previous numeric sequencing
• Inventory scripts can now define nested groups, just like the text file versions of inventory representations.

This release also includes numerous new flags to the various modules in Ansible, including:

• The ability to use the ‘raw’ module without a Python stack
• fixes to service detection, particularly on Ubuntu
• various improvements to the logic of the git module
• more data returned for the shell and command modules
• Various fixes around path expansion and relative path handling

The template engine also got a little smarter, making new variables like $template_name and $template_run_date available to annotate rendered templates.  Previously we had, and still have, variables like $ansible_managed that also help with this purpose.

The EC2 module has gotten a lot of love, in particular from our friends at Eucalyptus Systems and Seth Vidal from the Fedora Project. It can now deploy multiple simultaneous instances and now is powered by boto, an excellent EC2 python library.

Ansible has a major player in the open source configuration management and orchestration space too.   Just this week we’ve passed the 1000 star/follower mark on github and even more impressively were recognized by Black Duck Software as one of their 2012 “Rookies of the Year” projects.   Over 10,000 people have seen the Ansible webpage in the last month.  We’re currently one of the top watched Python projects on github this month too.  Exciting times, that I could not have predicted.

I think there’s a couple reasons for that.  (A) we are really liberating people by making it easy for them to handle hard to automate IT tasks and bringing some new capabilities that we previously not there before anywhere else, and (B) we have a really friendly, inclusive community that helps people of all skill levels and shares similar aesthetics about how to help each other.  We grow stronger from our collective knowledge.

Thanks very much to everyone who has made this release, and all previous releases possible — whether it’s contributing a documentation fix, adding a new module, fixing a bug, contributing a core feature, helping a fellow user on IRC or the mailing list, or helping spread ansible at a local meetup, this is very much a community project, made for everyone.

Contributors to this release with a change in git include: Daniel Hokka Zakrisson (79), Stephen Fromm (26), Michel Blanc (10), Dag Wieers (10), Tim Gerla (7), Seth Vidal (7), Jan-Piet Mens (7), Brian Coca (6), fdavis (5), Shaun Zinck (5), Gregory Duchatelet (5), Silviu Dicu (4), Norman J. Harman Jr (4), Lester Wade (4), Felix Sipma (4), willthames (3), afterburn (3), Lorin Hochstein (3), Les Aker (3), Junegunn Choi (3), Chelsea Robb (3), ron1 (2), jkleint (2), igor (2), Vincent Van der Kussen (2), Rodney Quillo (2), Pepe Barbe (2), Nandor Sivok (2), Jure Triglav (2), Juha Litola (2), Jayson Vantuyl (2), Brendan Taylor (2), Blair Zajac (2), vincent Van der Kussen (1), root (1), junyoung (1), diefans (1), davixx (1), Yeukhon Wong (1), Sergey Popov (1), Rune Kaagaard (1), Piotr Kweclich (1), Mark Theunissen (1), John Gosset (1), Jimmy Tang (1), Jeff Blaine (1), Javier Candeira (1), Gert Goet (1), Francis Besset (1), Francesc Esplugas (1), Christopher A. Snapp (1), Chris Berkhout (1), Bowe Strickland (1), Alexandre Bourget (1). 

While I would like to thank everyone individually here, I cannot.  However Daniel Zakrisson deserves special thanks for fixing all sorts of issues in record time and knowing my own code better than myself, all the while being a tremendous resource in helping people with questions on IRC.

This has been a fantastic first year for Ansible and I can only imagine the future will be even more exciting.  1.1 is going to bring “check mode” for all of you wanting dry run / comparison features, new options for handlers/notifiers, and many other things, many of which i can’t even anticipate.  

This last release was codenamed “Eruption”, following our Van Halen song themed release history.  It seemed fitting for a 1.0 and Ansible starting to get more mainstream.   A user suggested “Mean Street” for 1.1, and it is one of my favorite Van Halen albums, “Fair Warning”. 

Just as the CBS Sunday Morning show leaves you with pictures of little fluffy animals in the field and chirping bird effects, we too will leave you with a scene. I leave you with the chorus to Van Halen’s Mean Street.

It’s always here and now, my friend
It ain’t once upon a time

It’s all over but the shouting
I’ll come and I’ll take what’s mine

We’re searching for the latest thing
A break in this routine

Talkin’ some new kicks
Ones like you ain’t never seen

Experiences with Cobbler deploying Ubuntu Precise (12.04 LTS)

It’s been a long time since I’ve worked with Cobbler in any serious capacity.   While I encouraged the original Debian/Ubuntu support to happen, I never really used it for anything but Fedora, CentOS, and RHEL. So, when the very first thing I had to do for my new job was to fight some preseeds and set up a new Cobbler server, here’s a few things I encountered. This may be helpful to others doing the same.

apt-get install cobbler on Ubuntu Precise (12.04 LTS) installs Cobbler 2.2, not 2.4, out of the box.  This is mostly ok, though at minimum the “reboot” power management command is broken in this release.   I’m not really sure what else in 2.4 I wanted, but suspect LTS won’t update it.

As a nice surprise, the /etc/cobbler/settings file is generated by the package rather than packaged in a default state.  As a result, the initial ‘cobbler check’ command only reported one thing I needed to change, which to me was extremely well done, as a typical Red Hat install will not try to do things like fill in your hostname.  I still recommend reading /etc/cobbler/settings after installing to make sure it has what you want in it though.

Some initial confusion out of the gate — Ubuntu mislabels the /etc/cobbler template files for the preseed menu to be “Ubuntu Maas” even when MAAS is not installed. This is easy to fix by editing the template, but strikes me as an undesirable misappropriation of the project similar to the way I felt about Ubuntu Orchestra mostly being Cobbler-web.   I don’t like folks taking credit from the community for software they didn’t write.

cobbler import works mostly as advertised on the Ubuntu 12.04 LTS DVD ISO.   That was great.  Glad to see that going after many many years of seeing people ask questions about why that didn’t work as well as the Red Hat ones!   Awesome.  

Of course, cobbler just drives PXE, and you still have kickstart/preseed/autoyast to deal with.   Preseeds are nicer than AutoYAST (not XML), but WAY more painful than Anaconda kickstarts, due to needs of stringing together late commands and needing to be overly verbose about the answers to lots of questions.   Here’s a link to one work is using.   

On top of that, Precise 12.04 LTS exhibits a bug where the network interface set in the preseed is ignored and can only be set by kernel arguments.   Seems like it should have an option to just use the boot interface like Anaconda does.

“cobbler system edit —name=foo —kopts=”netcfg/choose_interface=eth1” fixes the problem

Finally, the installer doesn’t handle old LVM data well at all, requireing the following hack in the preseed that is, as far as I can tell, completely impossible for standard users to do manually without launching into the shell.

The fix is just:

d-i partman/early_command string vgs —separator=: —noheadings | cut -f1 -d: | while read vg ; do vgchange -an $vg ; done ; pvs —separator=: —noheadings | cut -f1 -d: | while read pv ; do pvremove -ff -y $pv ; done

The sad part is, it has to be all on one line.

Very minor — I noticed the logrotate file for cobbler was also not packaged, so I picked up the one from upstream so logs wouldn’t grow unbounded.   Still, it seemed like packaging was kind of quick and dirty in that regard and people didn’t really use it in production, or didn’t care about really huge log files.   I’ll admit the logging in Cobbler is not very good anyway, and in 2.0, it is pretty noisy and needs controls — even so, more need for logrotate.

Another minor packaging error was that cobbler-register wasn’t packaged, but some random NIH “cobbler-enlist” tool was instead, which was inexplicably a C program.  I am guessing this was used for MAAS at one point, but it seems misnamed, and I think all binaries packaged by the upstream package should be there.  Koan also didn’t seem to be available, but I didn’t need it — guests are going to be installed via openstack and I don’t need —replace-self because I have actual power management hardware.  I filed a bug about this as well as the above issues.

I’ll note that the Cobbler 2.2 default Ubuntu preseed was pretty bad and didn’t result in an automated install — this is an upstream thing, not a packaging thing.  I have higher hopes for the one packaged with 2.4, and am hoping pxe_just_once and all that just works — it would be much cooler if all the network templating stuff did as well — but if not, that’s ok.  Work has patched up our preseed pretty well to make things run all of the triggers and so forth.

In all, I think things worked pretty well and most problems lie with the Ubuntu installer — and Cobbler is able to manage pretty well, and the packaging for Ubuntu vs EPEL/Fedora rates a solid B.   App worked pretty darn well, really.

In all, I’m glad to see Cobbler in such a great state so many years later.  I don’t quite agree with the additions of the “config management” features in Cobbler (they are pretty basic and you will have to replace them pretty quick, plus don’t have good ways to manage them), but all in all, I’d still pick Cobbler again for a new project, no question…. and I’m happy to see Cobbler+Debian has gotten a lot easier to install.  Cross platform is important and a huge factor in doubling the size of your community.

Now if only Ubuntu and Debian would just switch completely over to Anaconda + kickstart, things would be much easier!

Thoughts on Approaches to Software Development

Why do we do the things we do?   If we think about what we like, patterns start to emerge.   Am I in software because I like typing curly brackets?   Maybe it starts that way, but it’s definitely not that.  In some essence, I like software because it’s some sort of abstract organizational logical space, essentially math — I like the design and planning — the code is fun but a means to an end.  But it’s more than that too.  I like it most because it’s a creative space and I am addicted to whiteboards and bouncing ideas off of people and seeing where those ideas go.  I like being surprised.

At a larger scale, I like to make things that cause ripple effects, that bring people together that wouldn’t otherwise work together.   I like to work with people that can teach me things, and enable others to bring ideas forward in ways they didn’t think they could before.   I am most inspired not by what I create, but what I see others evolve creations into, where things are used, and the things those things enable — in many cases much more impressive than what the individual cogs did themselves.   This the dynamic that I think I’m always searching for, and it sometimes can struggle to survive in industry environments, or even in personal projects.   We worry about small technical details too much, and miss the larger picture, or the experience and process of creation.

Some of the best parts in working in open source spaces are to see people I’ve worked with start other new projects, or come up with completely new ideas I didn’t expect, that play off my own.   Success criteria is not the software feature, but a byproduct.  Did your idea about how to make XYZ faster lead to a scientific discovery, or ultimately end up helping someone (or several thousand people) in some country you have never been to save 15 minutes on their daily commute?   Did your program play a really small part in that movie you really liked?   Do you have bits of code running on all the continents?  (passes given for Antarctica, that one is hard).

In earlier days as a developer I cared more about showing what I personally could do, lines of code, impressiveness of code, impact.   Ultimately, code is code, and if you look at code you wrote several years ago, you are hopefully going to hate it and realize you didn’t know a whole lot.   That code in the past doesn’t matter too much to your users or fellow contributors, nor should it matter much to you.  The most important contributions are your contributions to the flow of ideas, and only some of those come in code form.

The change you make in an organization is not what you personally execute, it’s what you enable everyone to execute on.   Good “agility” is not a project management idea or a way to get metrics, it’s the idea that anyone, anywhere, can make an idea that, if correct, can actually change the way you are doing everything.  It is the idea that no decision is sacred.   However, in many cases, it’s also knowing that some decisions are best made and moved on from, because there are more important decisions that need thinking over.   Getting stuck in a democratic discussion about a very small technical component is not agile either when it means you aren’t maximizing impact elsewhere — for your customers and users.

Similarly, we should share what we know with others, but we should not hide behind the impression of absolute conviction, if that would allow us to learn and evolve less.   Help transfer knowledge and everyone grows as a result.  The best customer and user relationships are the ones where you can learn as much from the customer or user as they can learn from you.   They are transformative.   And most of the complaints about the quality of your app?  Learn and grow from them.   Those first impressions are rarely inaccurate.   Occasionally, you’ll encounter bias, but realize everyone’s brains do work differently and it’s hard to please everyone.

When running or creating software projects, create places that breed ideas. To paraphrase the Lord of the Rings somewhat, it is dangerous when you create a project because you don’t know what it will turn into.  In any event, I think what you create is an environment where people create, rather than someone’s own exclusive vision.

Perhaps it’s systemic in Western software development (seems to be), but I think there’s an over emphasis in needing to be ‘right’ that we need to try to squash.   We need to learn not to over stress smaller parts of the equation and to think more about the larger parts.  We have to continually squash the idea that all of the good ideas need to be ours, or that we need to build something of our own because this other guy’s code didn’t do some crazy small facet of something.  Nope, that other guy was smart to!  He’s probably reasonable!  He probably accepts patches as well.   He might teach you something.

It also becomes easier as developers if we consider that the stream of ideas is our product, not just the code.   It’s often easy to get attached to code, and that’s a trap.

Teamwork in software contexts can be hard because if “Hackers and Painters” is an apt analogy, none of the European masters were that into collaborative painting — how do you hold your own creative unified vision in that vein?   Maybe it’s more like a group exhibition than a singular work.  As you work, resist the urge to paint over other people’s work (let them feel creative too!), look into APIs and design by contract, but also don’t view your code as something that must survive because it was your creation.

But what we are doing is not producing code, it’s more of a classical salon, where we are producing ideas.   Code is the absolute least interesting thing involved in what we create, and when we get bogged down in “software craftsmenship” we often loose focus about what the product really is, or what impact it really causes.

A summary of advice from this rambling post?  Maximize ripple effects, in ways that you can magnify others to do more.   Create frameworks that create accelerating forces.   Enable yourself to bend.   Learn from everyone and share what you know with everyone.   Learn where you were wrong before.   It’s much better than trying to be an awesome cyber ninja rockstar or what have you.   Judge your day by asking both “what did I learn” and “what did I share with someone”.

If we do that, SkyNet happens faster (it will have more help), and maybe, when SkyNet happens, it will like us.

Tips on Using Debuggers With Ansible

This blog post is intended to help some folks on the mailing list.  

Because we strive to keep Ansible simple, it can attract contributions from a wide range of people, who are learning Python, and then want to start doing more and more with Python.  This is great!  One thing that is not well covered by most Python resources is how to use the debugger, so I wanted to share this with everyone. With Ansible, there are also some specific things that go on you also need to be aware of when using the debugger.

Most people using python debuggers use “pdb”. This is a great debugger, but one we frequently use at work for Django and other things is “epdb”. The reason to use “epdb” over “pdb” is epdb contains a remote debugging feature that can sometimes be useful for debugging “over the wire” to processes where you don’t have console access, as opposed to just debugging foreground applications. You may still wish to learn pdb.

(What is super confusing is we’re talking about this epdb, not this.)

For starters, install epdb.   Pip will get the right one.

pip install epdb

Read More

Music Theory with Python

A while back I was working on a Ruby music theory library as a side project, and more recently heard about Mingus (a similar Python project) at PyCarolinas.  Mingus happens do most everything I was shooting for with my Ruby project, with the only downside being it’s not very object oriented, and the huge upside being it’s basically complete, where mine was not — though admittedly, I was doing it to explore algorithmic composition more than anything.

Here’s a pretty simple example of printing out a scale chart and the major/minor chords in the various scales.

If you are interested in Mingus, check it out on Google Code.

# simple demo of mingus -- http://code.google.com/p/mingus
# a python music theory library
#
# (C) Michael DeHaan, 2012
# LICENSE: public domain

import mingus.core.notes as notes
import mingus.core.diatonic as diatonic
import mingus.core.intervals as intervals
import mingus.core.chords as chords
import mingus.core.scales as scales

all_keys = diatonic.basic_keys
all_scales = [ 'diatonic', 'natural_minor', 'harmonic_minor', 'melodic_minor' ]

def ppc(chord):
    ''' print a chord nicely, without it looking like an array '''
    chord = [ "%-02s" % n for n in chord ]
    return " ".join(chord)

def simplify(chord):
    ''' I don't like reading things like C## and notes.remove_redundant_accidentals doesn't work '''
    if type(chord) == list:
        return [ notes.int_to_note(notes.note_to_int(n)) for n in chord ]
    return simplify([chord])[0]

# for all scales I want to print out
for scale in all_scales:
    
    # run the scale in all keys
    for key in all_keys:

       # print the header
       print "%s %s" % (scale, key)
       print "*****"

       # get the notes in the scale
       scale_notes = getattr(scales, scale)(key)

       # print each note, along with the major and minor chord
       # for each
       for note in scale_notes:

           print "%-15s | %-15s | %-15s" % (
               simplify(note),
               ppc(simplify(chords.major_triad(note))),
               ppc(simplify(chords.minor_triad(note)))
           )

       print ""
    print ""


 

Interfacer is an elegant way to write nested CLIs in Python

One of my apparent occupational hazards is the need to write a lot of command line tools.   I feel one of the best I’ve ever written was the one for Cobbler, because it was a multi level command line. This is not something that is provided by the stock command line libraries used in Python. Further, it was quasi-modular, in that I could extend it by adding my own modules to it.

You could do things like:

• cobbler
• cobbler distro
• cobbler distro add
• cobbler distro edit
• cobbler systems […]

By being “multi-level”, the CLI allowed for grouping of similar commands “editing a distro”, “adding a distro”, in a bucket of like command line items (distros, systems, etc).

When building a new command line tool for rPath, I wanted similar functionality, but to be able to release the core as a self-contained open source project. An example of why is that we can release a command line tool and allow a user to write their own extensions that make it still feel like a cohesive part of the larger command line.  Further, the next time someone wants to write a multi-level command line tool, they don’t have to start from scratch.

This work is now available on github as Interfacer.

Ansible 0.8 Released! Prepare for 0.9!

I’ve just finished releasing Ansible 0.8, which is, coincidentally about 8 months from the start of our project. Growth has been, to put it bluntly, astronomically defying my expectations. The need to get up early to attend to 10 pull requests from Europe before the morning compute is simply been an amazing experience. 

We’ve accumulated over 100 contributors, and are soon approaching 700 followers on github.  The mailing list is also huge and busy. Ideas are everywhere. It has been a wild ride. Thank you all for being a part of it.

0.8 brings a lot of features form a lot of hard work from many individuals.

While the full breadth is summarized in the changelog, a few things are worth pointing out.

Fireball Mode — This is a 0mq accelerated connection type. While Ansible manages by SSH by default, and requires no daemons to manage your remote systems, fireball mode uses ansible to distribute an ephemeral time limited daemon that offers EXCEPTIONALLY fast connections, all encrypted over AES using shared keys exchanged over SSH as a secure channel. For those willing to accept the additional temporary daemon, it flies like no other. Using this with —forks 50, and your infrastructure gets managed in no time. 

Playbooks can now import playbooks in other directories and those can import tasks in other directories relative to them. This allows for far greater options in terms of organization of how orchestration rules are defined.

There is now a “pause” module (courtesy Tim Bielawa) for waiting for a defined amount of time or until a user decides to proceed. Ansible is not just a configuration management tool, it can be used to run a wide variety of processes, and the idea that a human may occasionally be required is now well handled. In addition there are also some other modules added for managing INI files and crontabs. Ansible now allows pluggablity of “server side actions”, like pause, so users can also write their own server side actions.

There’s also now a “lookup plugin” facility, which is used to implement, as a first example, “with_fileglob”, which is a way of easily enumerating a local directory for transfer to a remote.   It’s great for templates, copying files, or transferring SSH keys. There will be many more modules using this facility in 0.9.

One of the behind the scenes things that many people won’t notice is that our module docs are now completely auto-generated, which was spearheaded by Jan Piet Mens. This is a great addition because it means when modules are improved or added, it doesn’t take a seperate pull request to update the docs project. This means documentation is always going to be up to date and consistent.  

Dag Wieers and Jeroen Hoekx have also started the ansible-provisioning project, which I have high hopes for.  If you have ideas for it, please stop by the mailing list.

And now, without further ado, Ansible 0.9 begins, codenamed “Dreams” (keeping with the Van Halen songs motif). Among the major goals for 0.9 include the new “when” operator, which will provide syntactic sugar around conditional operations (we already have “only_if”), continued performance enhancements, the evolution of the Ansible REST API, and lots of other shiny things.

Here’s a list of all the great people that helped bring you 0.8, “Cathedral”.  (Apologies again as gitstat massacres diacritics).

Jan-Piet Mens (43), Daniel Hokka Zakrisson (40), Dag Wieers (39), Tim Bielawa (37), Marco Vito Moscaritolo (27), Lorin Hochstein (18), Stephen Fromm (14), Wes Johnson (6), Peter Sankauskas (6), Dane Summers (6), Brian Coca (6), Nigel Metheringham (5), Matt Wright (5), Jeroen Hoekx (5), Aurelien Bondis (5), Seth Vidal (4), Matthew Johnson (4), Mark Theunissen (4), Luke Antins (4), Jonathan Palley (4), John Callender (4), Aleksej Romanov (4), Timothy Appnel (3), Serge van Ginderachter (3), Romeo Theriault (3), Michel Blanc (3), John Kleint (3), Jesse Andrews (3), Derek Carter (3), willthames (2), Philipp Grau (2), Pepe Barbe (2), Michael Lambert (2), Maxim Burgerhout (2), Matt Coddington (2), Dietmar Schinnerl (2), Chris Geddings (2), Ashley Penney (2), Ali Asad Lotia (2), mxxcon (1), Stijn Opheide (1), Rodney Quillo (1), Ralph Bean (1), Piotr Kweclich (1), Petetin Ludovic (1), Nathan A. Feger (1), Max Spransy (1), Matthew Williams (1), Matt Klich (1), Martijn Koster (1), Marko Mikulicic (1), Ludovic Claude (1), Lennie (1), Dave Coutu (1), Daniel Neri (1), Collin Allen (1), Alan Orth (1), ADRIyas (1)

Stay tuned.   Exciting things are happening in the world of systems management.