As this thread indicates, I am very glad to announce I’ve joined Reductive Labs as their new Product Manager.

As most folks on Fedora Planet know, killer OSS communities and taking over the computer world with intelligent software are some of my favorite things. Puppet excels at both of these, and Reductive Labs is full of amazing people. I simply cannot think of any better place to be right now. The present state of things with Puppet is great, and there are lots of places it can go.

It is also great to be seeing many of the same nics on the lists and IRC again, and I plan to be running into a lot more of you at conferences as well. If anyone wants to talk shop about their Puppet usage or management challenges, I’ll always be up for it. I’m mpdehaan on #puppet and michael@reductivelabs.com over email.

Gource is an amazing program for visualizing commit history in a git-based code project. What I like about it is that it can also show what areas of the project are active in an easy to understand way, to show whether there is community around a whole project or just aspects of it. What looks like a shiny useless visualization is, in fact, pretty useful stuff. I’ll get to that in a bit.

So, I needed something to scan and past OSS things I’ve been involved with were logical first targets. To index the history on Cobbler into a concise video, I ran the following:

gource -s 0.03 --auto-skip-seconds 0.1 --file-idle-time 500 --max-files 500 --multi-sampling -1280x720 --stop-at-end   --output-ppm-stream - | ffmpeg -y -b 3000K -r 24 -f image2pipe -vcodec ppm -i - -vcodec mpeg4 gource.mp4

Want to run this yourself? You will likely have to build gource from source. I’ll warn you that building from source involves installing a ton of deps, though all are in Ubuntu 9.10, and once ./configure finally passes it does build fast. Fedora was downlevel with respect to ftgl, and compiling ftgl from source was difficult, hence the Ubuntu usage. The parameters I use above result in a large video (75MB) but are intended for YouTube HD.

The result is below. Note that I didn’t keep my source control commit attribution for the first couple of years on the project (lesson learned in how to use git!), I used to do development on devel and switched to master (this video shows master), and koan was grafted into the cobbler tree late in the game. Early on, I committed from two different user IDs. As a result, the video is not perfect — things tend to “pop” into view as releases happen. You’ll see the first outside attribution happen about 1/2 way through, though of course this was happening much much earlier. Still, the acceleration at the end, I think, means we achieved something pretty decent. Not all projects do.

Perhaps this is a start of a good meme. Get your code up on YouTube. Show us the life of your code and who you collaborate with.

Watch

I can see gource being immediately useful for a one major purpose. When evaluating OSS software for use in business, you always need to know if the community is solid and self sustaining. This allows you to watch a short video and find out. Coupled with looking through the mailing list archives, that’s a pretty good check. It can also help identify interesting patterns of large scale refactoring, new development, or stagnation.

Gource may also a great way to explain open source to people who don’t immediately understand how collaboration can work, and how contributors come and go.

That is what I call good TV. It is also rather trippy to look at. Please turn up the Floyd.

If we had a free supercomputer and infinite development time, my ultimate dream visualization would be a 20×20 foot wall section of these graphs, showing multiple projects side by side, with developers flying between projects. Who flies between projects? Is that common? What are the clustering patterns of these projects that share contributors? Where are the hubs and spokes? Are the hubs bigger projects than the spokes? (Can we get that in 3D?). There is something to be learned here, even if we don’t know what that is.

Install Gource. Let’s see your project video.

I was looking for a simple way to diff two complex hashes. In order to make the output nicely readable for humans, I’d first like to flatten the hashes. For example:

test = {
         "a" : [ "dog", "cat", "chicken" ],
         "b" : {
             "c" : 0,
             "d" : [ "red", "yellow", "blue" ],
         },
         "e" : "shiny"
   }

Becomes:

{
    'a': ['dog', 'cat', 'chicken'],
    'b.c': 0,
    'b.d': ['red', 'yellow', 'blue'],
    'e': 'shiny',
}

Here is a very long Perl module that does this. Here’s my cut:

  def _flatten_ds(self, ds, result=None, memo=""):
        if result is None:
            result = {}
        assert type(ds) == type({})
        for (k,v) in ds.iteritems():
                if memo == "":
                    new_memo = k
                else:
                    new_memo = "%s.%s" % (memo,k)
                if type(v) == type({}):
                    self._flatten_ds(v, result=result, memo=new_memo)
                else:
                    result[new_memo] = v
        return result

Almost 300 lines shorter than the Perl module

Today I’ve reworked LookAtGit which was a project I originally implemented because (A) I was really bored, and (B) I wanted to learn Scala.

The new “v2″ is done in Ruby, which is nice because the regexes there don’t make me want to hurt someone and Ruby is generally awesome, and I’ve forgotten how much I missed it having a lot of corporate Python jobs. Ruby is fun.

So, lookatgit… I’ve also optimized it a HUGE amount since last time, for one — I found “git log –shortstat”, it has to execute a billion less commands and can also skip binary commits.

The rewrite doesn’t quite yet offer some of the statistics-oriented statistics (SOS) that the previous version offered, but it does offer some additional reports, arbitrary field sorting, and will enable adding lots more other reports in the future.

To get started, check out from github and read the README file in the “v2″ directory here.

You can see from those instructions how to do simple things like see the top 50 most active files in a repo, or the top 100 contributors sorted by arbitrary statistics.

Here’s an example from Spacewalk. Spacewalk is a HUGE repo, and while I’m limiting the length of the output below, the report time is spent in the calculations, rather than the output, so with a reasonable machine it will only take a minute to scan the repo.

mdehaan@snowball:~/code/lookatgit/v2$ time ruby lookatgit.rb -r ~/code/spacewalk/ --limit 10 -T -F --header --verbose
scanning...
processing 9534 commits...
generating report...
--------------------------------------------------
TOP CONTRIBUTORS REPORT
name,lines_changed,lines_added,lines_removed,commit_ct
--------------------------------------------------
Miroslav Suchý <msuchy@redhat.com>,1587958,143182,1444776,1650
Jan Pazdziora <jpazdziora@redhat.com>,183227,121126,62101,1428
Michael Mraka <michael.mraka@redhat.com>,196978,94669,102309,781
Devan Goodwin <dgoodwin@redhat.com>,146951,44522,102429,611
Justin Sherrill <jsherril@redhat.com>,1398046,763492,634554,587
Pradeep Kilambi <pkilambi@redhat.com>,726023,361122,364901,476
Mike McCune <mmccune@gmail.com>,39470,27793,11677,447
jesus m. rodriguez <jesusr@redhat.com>,392192,112554,279638,430
Partha Aji <paji@redhat.com>,49294,29033,20261,406
Milan Zazrivec <mzazrivec@redhat.com>,42101,22507,19594,375
------------------------------------------
TOP FILES REPORT
filename,lines_changed,change_ct,author_ct,commit_ct
------------------------------------------
java/spacewalk-java.spec,2718,246,17,246
backend/spacewalk-backend.spec,2420,235,14,235
java/code/src/com/redhat/rhn/frontend/strings/jsp/StringResource_en_US.xml,23742,202,19,202
java/code/webapp/WEB-INF/struts-config.xml,8040,147,18,147
rel-eng/packages/spacewalk-java,291,146,13,146
web/spacewalk-web.spec,1739,139,11,139
java/code/src/com/redhat/rhn/frontend/strings/java/StringResource_en_US.xml,8999,128,14,128
schema/spacewalk/spacewalk-schema.spec,726,121,10,121
rel-eng/packages/spacewalk-backend,237,119,11,119
proxy/installer/spacewalk-proxy-installer.spec,533,99,7,99

real	1m1.190s
user	0m6.912s
sys	0m0.340s

The next step is to build in those “statistics oriented statistics” and enhance the query capabilities. For instance, I’d like to be able to generate a report on the standard deviation times between commits, to show which developers on a given project are slacking off . Similarly, I’d like to generate aggregate statistics on a project so I can show that Project X contributors typically commit changes with certain distribution patterns, which may or may not be revealing.

Contributors are very welcome. I currently do not have a project list, but if enough folks are interested we can get this going. It is not so much about what it can generate now but what we can generate in the future.

Do you get excited about new language features? Probably. Can you use them immediately at work? Sometimes. If you write software that you distribute though, you often can’t!

Wouldn’t it be awesome if I could get all of the Python Package and Ruby gems (and CPAN) as RPMs on EL 4 and I get to pick my interpreter version (any version I like) and be able to choose from multiple interpreter versions to run on the same system?

Part of the problem with writing software that you want to be easy to set up and install for users is that you can’t use the shiny newness. For instance, if you have to support EL 2, you won’t be excited about new features in Python 2.5, as you’ll never be able to use them. Same deal with Rails 3 or TurboGears 4000.

If you are a hosted service, you could decide to do a lot of work and become a mini-distribution (packaging these things for yourself and tracking security updates and other bugfixes), but that’s a lot of duplicated IT across the world for everyone trying similar things. Each new library you want to use becomes a discussion with IT because someone needs to package it and look after updates. Ouch!

I am not a fan of the java-style mode of deployment as it seems to imply a great chance for security vulnerabilities (due to lack of updates by packaging a sub-module with your code), retards progress, and also tends to encourages forking. However I can kind of understand why it occurs. Fear of the outside world breaking your code, or not being able to deploy what you want where you want it.

Virtual appliances are also the wrong answer to deployment, because of the same update concerns, and the fact they take a sledgehammer to the problem and waste resources.

Ideally what I think I want is a cross-distribution build server that all upstream software projects could use that would automatically build packages for different interpreter versions and distributions. By cross distribution I not only mean all of Fedora, CentOS, and RHEL, but also Debian and Ubuntu. If we are lucky, also OS X. I’m tired of OS X deployments working differently.

Then I should be able to do:

yum install python25-simplejson python24-simplejson

and run the same interpreter on the same box.

Some issues are to be had with contention over Apache, I’m assuming, though I think it would be really awesome if we could take every hosted service developer in the world out of having to maintain their own libraries when Enterprise Linuxes are too far behind and the likes of Fedora or Lefty Lemur are too fast-changing and unstable.

I don’t forsee this happening, of course… but you know… sometimes I think it would be nicer if the newness was easier to deploy on the oldness. To do this though, we really have to take the human distro-specific packagers out of the equation, and make a build service that is very very encouraging for all upstream developers to use. And it should (because of the upstream focus) ideally involve a partnership between distributions. This may also require unifying Debian and RHEL packaging in order to gain widespread adoption of software developers packaging their own content and submitting it to common build servers. If possible, do OS X as well.

Also, I want a pony.

On Christmas Eve Eve Eve, I bring to you to the greatest software carol about web client/server communication ever composed to date. I know because I spent like 15 minutes on it.

Use REST, you Merry Gentlemen

Use REST you Merry Gentlemen
Let nothing stateful stay
Remember Hypertext our Saviour
Gets out of the firewalls way
To save us all from SOAPs confusion
When we were gone astray
O tidings of interoperability and joy,
Interoperability and joy
O tidings of interoperability and joy

In Irvine, in Cali-forn-ia,
This methodology was born,
And shared among the intertubes,
The W3C did morn.
To which the Enterprise crowd
Did rather take in scorn
O tidings of simplicity and joy,
simplicity and joy
O tidings of simplicity and joy

From code to our ed-it-ors
The aforementioned practice came;
And unto certain code-o-ers
Brought tidings of the same:
While not quite well defined as XMLRPC
It works almost the same.
O tidings of interoperability and joy,
Interoperability and joy
O tidings of interoperability and joy

Now to fast JSON sing our praises,
All you within this place,
And with true love and brotherhood
Each language it does embrace;
This great tide of options;
Lock in now replaced.
O tidings of interoperability and joy,
Interoperability and joy
O tidings of interoperability and joy

I’m reclaiming “Pragmatic” from Dave Thomas et al. He doesn’t get to own it

Things I would like to see more of among OO Software Developers:

• OO design that is about message passing and accurately modelled domain objects, not setter/getter boilerplate, which is actually quite procedural, and breaks modeling, nor abstract objects that don’t accurately model the problem they are solving.
• Design Patterns being used in the sense that Christopher Alexander intended them, as a way to talk about code, not as a LEGO kit. Don’t use a Factory to replace an if conditional if you’ll only use it once. Don’t use a Command to replace a function call. Queue the popular expression: If all you have is a hammer, good luck removing those lug nuts. Read this.
• More Of the Unix Philosophy. Small components that can talk to each other are easier to understand (think find, pipes, and xargs). Black boxes are our friends. Well defined interfaces between modules help us build those boxes.
• YAGNI being exercised as a first principle with regard to abstraction layers that are probably not going to be needed. (Exception: databases). Write code that does something useful today, not something that plans for an uncertain future. If the code is small enough, that future can be quickly dealt with later.
• Love of small codebases in general. You can do amazing things in under 20k of code. Taking pride in how little code you write, not how much code you write. Keep It Simple. Simple is awesome. Clever is usually bad. (Fun, But … Bad)
• More hybrid OO functional/development. Side effects can often be eliminated, and lead more to that message passing idea and better traceable code.
• More truly modular software. Almost every component of your software system should be confined to a small enough area to be replaceable. It should not be abstracted out as a first principle, as that is just added work…. but you should know that it would be possible to replace it in the future. Investing heavily in any particular framework limits future options, especially when the framework starts to limit technology decisions. It is for this reason that I really like the concept of language agnostic message buses, or at least XMLRPC.
• More Use of Dynamic, Duck Typed Languages. Static typing errors don’t actually happen much in practice unless someone fails to write functional tests, and even then don’t happen very often. They can also be avoided with simple asserts, just ask if the “Duck” is able to quack or not, not whether something is a Duck. Such languages are not only more fun to work with, one person can do the work of five Java developers.
• More Use of Other JVM Languages. If interacting with Java libraries is a requirement, more use of Groovy, Closure, or Scala can go a long way to allow folks to be more productive and to write more sustaintable codebases. Not enough people know they have options, which tends to perpetuate some bad design decisions. The Java platform tends to require (unfortunately) components that also speak Java standards, because the community isn’t so interested in truly open standards that give all languages the same footing. Why is that a problem? Just as Perl isn’t REALLY that horrible of a language, it is just that the best practices surrounding Perl are not so good usually, Java best practices tend to favor overdesign that eventually leads towards unmaintainable and impenetrably dense software. Being able to choose languages that enforce better design practices (when you are forced to use something Java compatible) can help avoid this slippery slope.

Yes, I just said that using Java leads to unmaintainable software. You may print that.

 ________________________________________
< it looks like you are writing a letter >
 ----------------------------------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Cobbler leveraged JSON for saving object representations to disk (though not for wire transport, where it used XMLRPC, but that is another story). The object representation system in 2.0, which I called “FIELDS” was used to define the members of each object was a little more heavy weight than I liked. This made it difficult to add new “nouns” into Cobbler, and made modelling system interfaces (which I didn’t want quite to be their own objects), rather hard. Why? The serialization and de-serialization to and from objects was not as recursive as it could be. We also had a top level orchestrator object called “config” that did not do much, that I always wanted to get rid of.

I have.

Here is a rough example of doing all of that more automagically.

The base class is /somewhat/ gnarly because of the automagic, but I don’t think the object classes could be any simpler. Notice the type checking that is also built in. Bonus! So you can work in JSON objects all day, but your code gets to work in /objects/, not error prone nested hashes.

Extra validation can be added by writing setter functions, which are, by default, optional .. you get free type checking without the setters.

[github.com]

There’s not much to say beyond the documentation examples, but I was looking for a way to do streaming uploads from Python, and poster delivers nicely on the client side.

Writing a server side streaming downloader in mod_python is infinitely easier and looks something like this:

   def handle_upload(self):
       self.form_data = util.FieldStorage(self.request)
       for key in self.form_data.keys():
           value = self.form_data[key]
           if type(value) == types.InstanceType:
               if isinstance(value, util.StringField):
                   pass
               elif isinstance(value, util.Field):
                   filename = value.filename
                   content_type = value.type
                   base = os.path.basename(value.filename).strip()
                   fd = open("/tmp/apache/%s\n" % base, "wb")
                   while True:
                       data = value.file.read(1024)
                       fd.write(data)
                       if data == "":
                           fd.close()
                       break
       return apache.OK

Security and error handling (as well as writing the rest of the mod_python handler, however minimal that might be) are left as an exercise to the reader.