My first thought was just to search for help on Spotlight, and that's not a bad starting place as the page covers a number of keywords, more correctly called metadata attributes. Spotlight's help covers kind, author, date, created and by as well as the boolean operators AND, OR and NOT. But I knew there were many other metadata fields that were commonly used in files such as images and audio files. I had come across the mdls shell command before which lists the metadata fields on a file. A quick check of a JPEG image revealed all kinds of interesting data:

da-imac-01:stuff  david$ mdls IMG_3564.JPG 
kMDItemAcquisitionMake         = "Canon"
kMDItemAcquisitionModel        = "Canon EOS 10D"
kMDItemAperture                = 0.970855712890625
kMDItemBitsPerSample           = 32
kMDItemColorSpace              = "RGB"
kMDItemContentCreationDate     = 2008-04-22 18:40:25 -0400
kMDItemContentModificationDate = 2008-04-22 18:40:25 -0400
...
kMDItemFlashOnOff              = 0
kMDItemFNumber                 = 1.399999976158142
kMDItemFocalLength             = 50
...

This is a truncated list of the 50+ fields in one of my image files. Note some of the interesting ones like the aperture, flash setting and focal length.

I played around with another of the Spotlight/metadata shell commands: mdfind. This lets you do the equivalent of a Spotlight search from the command line and after a bit of trial and error, guessing the keyword names and value formats was fairly easy:

da-imac-01:stuff david$ mdfind make:canon focallength:50 flash:0 iso:125
/Users/david/Desktop/Turks, April 2008/IMG_3564.JPG
/Users/david/Pictures/iPhoto Library/Originals/2008/Turks, April 2008/IMG_3564.JPG
/Users/david/Pictures/iPhoto Library/Originals/2008/Museum Visit/IMG_3273.JPG
/Users/david/Pictures/iPhoto Library/Originals/2008/Museum Visit/IMG_3276.JPG
/Users/david/Pictures/iPhoto Library/Originals/2008/Mar 23, 2008/IMG_3288.JPG
...

Peeling the Onion with DTrace

Although these shell commands are very useful, the man pages for the commands do not list the valid search keywords. I knew there must be a list of the keywords used in Spotlight search bar that mapped to these constant names so I thought *what a great time to learn dtrace!*

For those of you who haven't heard of dtrace I encourage you to play around with it. It's a very powerful tool for doing live probing and tracing of low level activity in the operating system. After skimming this nice tutorial I tried this command in one window:

da-imac-01:bin david$ sudo dtrace -n 'syscall::open*:entry /execname == "mdfind"/ \
    { printf("%s %s", execname, copyinstr(arg0)); }'
Password:
dtrace: description 'syscall::open*:entry ' matched 3 probes

and then ran my mdfind command again in another Terminal window. The dtrace "script" says to trace all system calls whose name begins with "open" when the system call is entered, but only if they were called from the mdfind process, and then print out the name of the system call and the first argument (which in the case of open is the file or device name). That resulted in LOTs of calls like this showing mdfind opening all kinds of metadata importer files, which I assume are libraries that know how to manipulate certain types of metadata attributes:

CPU     ID                    FUNCTION:NAME
  0  18390              open_nocancel:entry mdfind /System/Library/Spotlight/\
    Audio.mdimporter
  0  18390              open_nocancel:entry mdfind /System/Library/Spotlight/\
    Audio.mdimporter/Contents
  0  17604                       open:entry mdfind /dev/autofs_nowait
  0  17604                       open:entry mdfind /System/Library/Spotlight/\
    Audio.mdimporter/Contents/Info.plist
  0  18390              open_nocancel:entry mdfind /System/Library/Spotlight/\
    Chat.mdimporter
  0  18390              open_nocancel:entry mdfind /System/Library/Spotlight/\
    Chat.mdimporter/Contents
...

But the part of the trace I was most interested in was near the very end:

...
  1  17604                       open:entry mdfind /System/Library/Frameworks/\
    CoreServices.framework/Versions/A/Frameworks/Metadata.framework/\
    Resources/MDPredicate.plist
  1  17604                       open:entry mdfind /dev/autofs_nowait
  1  17604                       open:entry mdfind /System/Library/Frameworks/\
    CoreServices.framework/Versions/A/Frameworks/Metadata.framework/\
    Resources/English.lproj/MDPredicateKeywords.plist
  1  17604                       open:entry mdfind /dev/autofs_nowait
  1  17604                       open:entry mdfind /System/Library/Frameworks/\
    CoreServices.framework/Versions/A/Frameworks/Metadata.framework/\
    Resources/English.lproj/schema.strings
...

Note the files MDPredicateKeywords.list and schema.strings in the /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Resources/English.lproj folder. I tried looking at the schema.strings file but it was in a binary format. So I tried open schema.strings and sure enough, Xcode launched and loaded the file which contains over 400 lines of mostly metadata keyword definitions. The ones we are interested in are near the end and of the form kMDItemXXX.ShortName = yyy:

"kMDItemPixelHeight.ShortName"                = "pixelheight,height";
"kMDItemPixelWidth.ShortName"                 = "pixelwidth,width";
"kMDItemWhiteBalance.ShortName"               = "whitebalance";
"kMDItemAperture.ShortName"                   = "aperture,fstop";
"kMDItemAudioEncodingApplication.ShortName"   = "audioencodingapplication";
"kMDItemComposer.ShortName"                   = "composer,author,by";
"kMDItemLyricist.ShortName"                   = "lyricist,author,by";
"kMDItemStarRating.ShortName"                 = "starrating";

These are just a few of the dozens of entries to whet your appetite. For the most part, I've found them to work as expected, with one exception: starrating. I never got any hits using it so I tried using mdls on an MP3 that I knew had a rating set in iTunes and there was no metadata attribute set on it for the iTunes rating. So I guess all you Mac developers out there should "do as Apple says, not as Apple does."

Satisfaction

One of the things that I really like about OS X is the ability to work with the system at varying levels of depth. This diversion started with me playing around with the Spotlight search bar: a very advanced "desktop search" feature found only in the most modern operating systems. But when I wanted to learn more, I was able to easily muck around at the command line and experiment with the very same infrastructure that Spotlight is built on. Finally, I was able to leverage a very powerful, low level system tool, dtrace to probe the details of what was going on inside OS X which led me to the answer I was looking for.

Objects and Relationships

Things' data file is a fairly simple XML file that is primarily a collection of object elements. These elements contain attribute elements which are the properties of an object and relationship elements which can model a one-to-one or a one-to-many relationship to other objects in the file. Here is a snippet of a test file I used.

<object type="TODO" id="z159">
    <attribute name="focustype" type="int32">131072</attribute>
    <attribute name="focuslevel" type="int16">0</attribute>
    <attribute name="datemodified" type="date">227707669.31836900115013122559</attribute>
    <attribute name="datecreated" type="date">227707452.24475499987602233887</attribute>
    <attribute name="title" type="string">Todo 1.1.1</attribute>
    <attribute name="index" type="int32">0</attribute>
    <attribute name="identifier" type="string">7F63B75E-11F4-4153-B222-7506882CAD79</attribute>
    <attribute name="compact" type="bool">1</attribute>
    <relationship name="parent" type="1/1" destination="THING" idrefs="z169"></relationship>
    <relationship name="author" type="1/1" destination="COWORKER"></relationship>
    <relationship name="delegate" type="1/1" destination="COWORKER"></relationship>
    <relationship name="focus" type="1/1" destination="FOCUS" idrefs="z150"></relationship>
    <relationship name="recurrenceinstance" type="1/1" destination="TODO"></relationship>
    <relationship name="recurrencetemplate" type="1/1" destination="TODO"></relationship>
    <relationship name="scheduler" type="1/1" destination="GLOBALS"></relationship>
    <relationship name="children" type="0/0" destination="THING"></relationship>
    <relationship name="tags" type="0/0" destination="TAG" idrefs="z130 z102"></relationship>
    <relationship name="reminderdates" type="0/0" destination="REMINDER"></relationship>
</object>

You can see the relationship with the name parent which has one ID in its idrefs list. Further down, you can see the tags relationship has two references. There are three primary types of object elements:

1. TODO: an actual todo item
2. TAG: a tag object
3. FOCUS: these are a generalized object type used for Projects, Areas and groupings of these (like Today, Next).

All I do to parse the document is to create a dict for each object, using the attribute and relationship sub-elements as fields in the dict. Relationships are initially stored as lists of the string idref values. Then at the end of the parsing method, once I have all the objects loaded into an ID map, I resolve the references to the actual dict objects. This leaves me with a "Pythonic" graph of the data rather which is easier to work with (IMHO) for querying and processing.

def parse_things_xml(database):
    """Parse object nodes of from of XML object elements
    and save in dicts. Parses attribute elements for dict fields
    and follows parent/children relationship elements to link up
    related nodes. Returns a list of root nodes (those without
    parents).
    """

    def parse_relationship(relationships, name):
        """Parse the specified relationship out of the list of relationships.
        Assumes there is only one relationship of the specified name.
        """
        rels = [r for r in relationships
                if r.attributes['name'].value == name]
        if not rels:
            return None
        idrefs = rels[0].attributes.has_key('idrefs') and \
                 rels[0].attributes['idrefs'].value.split(' ') or []
        return idrefs

    xmldoc = minidom.parse(database)
    objects = xmldoc.getElementsByTagName('object')

    # parse each object element into a dict storing attribute
    # child elements as dict fields and child relationship
    # elements as lists of idref values
    node_map = {}
    for obj in objects:
        node = {'id': obj.attributes['id'].value,
                'type': obj.attributes['type'].value}
        attrs = obj.getElementsByTagName('attribute')
        for attr in attrs:
            val = attr.hasChildNodes() and attr.firstChild.data or None
            node[attr.attributes['name'].value] = val
        rels = obj.getElementsByTagName('relationship')
        for rel in rels:
            relname = rel.attributes['name'].value
            node[relname] = parse_relationship(rels, relname)
        node_map[node['id']] = node

    # resolve idrefs by replacing their value with a reference to the
    # actual dict
    for node in node_map.values():
        for relname, idrefs in node.items():
            if isinstance(idrefs, list):
                relnodes = [node_map[idref] for idref in idrefs]
                if relnodes and relnodes[0].has_key('index'):
                    relnodes.sort(key=lambda n: n['index'])
                node[relname] = relnodes

    return node_map.values()

I also have a method to query the object graph based on some simple selection criteria and then a ''very'' simplistic printing routine to print out the results. The code is a little rough, but it's a start. I've organized it into two files: thingslib.py which does the parsing and querying, and things.py which is a script that you can call to select the items of interest and print them out. The code is available from my download page.

So when I was putting together a playlist for a party the other night, I was just about to click in iTunes when I thought "why not try out Amazon.com?" I'd heard they had a good collection, all DRM-free, with equal or better pricing than iTunes. And you can usually count on Amazon.com to make the buying experience simple and smooth. So I surfed over to Amazon.com, searched for the Psychedelic Furs "Love My Way" and bought my first MP3 on Amazon.com.

The first time you purchase an MP3 on Amazon.com, it installs an application that manages the file download process and automatically adds the files to your iTunes library. The files are fully tagged and include artwork just like a song you would by from the iTunes Music Store. Additional purchases have a very similar experience to iTMS: search for a song, play a sample of it if you like, one click to buy and in a few seconds it's in iTunes.

Amazon's MP3s are also pretty high quality, encoded at or near 256 kbps, some using variable bit-rate and others using constant bit-rate. I also noticed that they use the LAME 3.x encoder, considered by many to be one of the best MP3 encoders out there.

In all I bought 10 singles and one 13-track album, spending a total of $17.61. Of the 10 singles I purchased, 8 were $0.99, and 2 were $0.89. The album was $7.97. Had I purchased these on iTMS I would have paid an additional $2.22 or 12.5% more (all of the songs were $0.99 on iTMS and the album was $9.99, and interestingly enough none of the songs I bought were available in iTunes Plus DRM-free format).

So not only did I save a couple of bucks, but more importantly I purchased DRM-free music where I'm not limited on how and where I can play my music. And the purchasing experience was about as close to a fully-iTunes-integrated solution as you could get without the software being written by Apple. Apple may need to swallow a bit of it's pride and go back to the record companies to work out DRM-free deals with everyone or they may be in for a tough fight with Amazon.com over digital music.

So then you’re off looking at parsing tools and trying to decide which one best fits your problem, but since all the simple ones have really only been designed to parse math expressions and all the fully-featured ones were really designed to parse the language they were written in, you come to that same conclusion again: you could probably just design a simple parser generator for reporting grammars on your own. Really, how hard can that be?

And then hopefully, you stop. And think. And try to wind your way backwards along the thread in your mind that led you down this path back to ... what was I trying to do? Oh yeah, create a simple report.

So that’s how I ended up writing a blog from scratch. And how I spent much more time building, tweaking and maintaining the site then I spent writing posts for the site. Which is precisely how I ended up with this new version of my blog, using RapidWeaver. It allows me to easily post blog entries, share photos and videos, but still gives me a lot of control of the site without turning this into a programming project.

That doesn't mean I won't do some hacking of RW (which I've already done, and I'll talk about more later), but I'm hoping I won't get sucked in like I did before.

The first problem arose after I had transferred over all my applications, data and settings from my Macbook. Apple includes a Migration Utility to help you move data from another Mac when you first setup a new one, and I thought I'd give this a spin and see how it worked. I chose to transfer over everything just to see what would happen. At the same time this was going on, OSX was downloading a number of OS and firmware updates to bring my machine up to date. I let both of these run to completion and then was prompted to restart my computer to apply some of the OSX updates (yes, updating a Mac sometimes requires a restart, just like Windows).

The machine restarted and I got the familiar login window, clicked on my name, entered my password and ... the screen went black. Huh? No, it wasn't completely black, there was a little text way up in the upper left corner::

Welcome to Darwin!
Login:

Somehow the OSX window manager had exited and dumped me right to the Darwin console. I tried logging in and exiting, rebooting, even typing Ctrl-D at the Login prompt to exit out of the terminal mode. Everything eventually just took me back to the graphical login window and from there, the console login prompt. Yikes!

Googling didn't reveal anything obvious so I did what any well-trained Windows user would do: I reinstalled OSX from scratch. But this time I skipped the Migration Utility, partly because I was suspicious that it may have been the culprit, and partly because I didn't really want to transfer everything over (and lastly because I'd never installed OSX from scratch and wanted to give it a try). That seemed to do the trick and the machine was now logging me into OSX correctly.

DVDead

The second problem I ran into had to do with playing a DVD. I play DVDs on my Macbook fairly often, and it works great. I wanted to see how one would look on a 24-inch display so I popped in Skin and Bones, waited for the usual whirring and humming of the DVD drive, saw QuickTime/iDVD come up and then to my surprise saw the message: Supported disc not available. After trying a few things (reinserting the DVD, using a different DVD, rebooting), I started to get a little worried.

This time, Google was my friend. I got quite a few hits about this problem and a pretty consistent suggestion as a solution: use the Disk Utility to fix the permissions. So I booted off the installation disk, ran Repair Disk Permissions (while wondering to myself, why is this necessary?) and restarted the iMac. I popped in the DVD and it worked.

Yes, the Mac is a great machine and OSX is a very nice operating system, but it does have its occasional problems. What's interesting, though, is that when I run into these issues on the Mac, I'm generally surprised. When I run into them on Windows, however, I usually just take them in stride, as if reinstalling apps, rebooting, and Googling your way out of the error tarpit is just the status quo. Using a Mac is like driving down a smooth highway and hitting the occasional bump in the road. Using Windows feels more like you are driving down a dirt road: you expect a rough ride and look forward to the occasioanal clear patches.

Opening any Apple product is an experience in itself. I sometimes think they put more though into the packaging of their products than many companies put into the actual product itself. Then once you unwrap the product, there are those first few minutes where you marvel in the simple, elegant and clever design and engineering. It doesn't seem to matter if its a monstrous 24-inch iMac, a sub $100 iPod or even a wireless access point. Opening a box that says "Designed by Apple in California" puts a smile on my face and brings back the same feelings I had as a kid when opening a Christmas present. (But now my toys are a lot more expensive).

Hardware

The iMac is very clean: it has a flush-mounted power cord that when inserted seems to emerge right out of the computer's case. There is a guide hole in the back of the stand to lead the cord through so it is mostly hidden from view. The various I/O ports are hidden behind the lower right of the display, leading cables back and away (rather than out the side of the display where they would be more noticable). The large power button is in the back of the lower-left corner of the display, keeping it hidden, but easy to find by touch.

The display is properly balanced on a hinge in the stand so even though its large and heavy, it takes very little effort to tilt the display when adjusting it. This is easily accomplished with one hand pushing or pulling on the lower edge of the display. There is just the right amount of friction to keep it in place but allow for fine adjustments without jerking the display.

And then there is the remote control: simple and small, but it does what you need it to when driving Front Row. When I opened the accessories box, I thought to myself "I probably won't use the remote that much but I don't want to lose it. Maybe I should just leave it in the box." Yep, someone at Apple thought about that too. That's why there is a magnet inside the right edge of the display so you can just stick the remote right on the side of the display. Its out of the way, but easy to find when you need it. Nice.

The display is beautiful. A 24-inch widescreen may not sound that big these days when 20-inch monitors are commonplace and many people have two or even three monitors, but when you are sitting a foot and a half away from this display it feels very large. I actually found myself initially having to move my head a little bit to see the different corners of the screen. I have 2 side-by-side 20-inch LCD displays at work but there's something about one complete workspace that feels a lot bigger. Two separate displays feel like, well, two separate displays. The borders of the monitor break up the display surface so you rarely want one window to span across them. It takes constant fiddling to keep them exactly the same height such that the two screens line up correctly, and frequently there will be slight color variations between one and the other that is both annoying and distracting.

The colors are rich and the display is very bright. And watching a DVD felt like I was really in a theater (or at least in front of a very nice TV). This display covers so much of your peripheral vision that its easy to get lost in the movie and forget you are watching it on a computer. I never had that feeling on the Macbook because everything around the smaller display was constantly in my attention zone and can easily become a distraction. This display really grabs your full attention.

Note that while I was very impressed with the out-of-the box experience of my new iMac, I did, however, run into a few snags with the new machine, which you can read about here.

Where are the Beautiful PCs?

As I was playing with my new iMac, I kept wondering why can't you buy such a machine from a PC manufacturer? Apple now has the price of their line of computers competitive with equivalent PCs, so its not like they would be so expensive that people wouldn't buy them. And its not like people don't appreciate good hardware. People pay a premium for all kinds of specialties: size with ultra-portables, performance with high-end gaming machines, quiet PCs, small form PCs, PCs with large displays. So wouldn't it make sense that in such a large market there would be a segment of people who would pay a premium for a beautiful, well-designed, all-in-one PC that is simple to setup, just works and is a pleasure to use? Of course, with Bootcamp, Parallels, and now VMWare, there is. The iMac.

There is no best programming language, contrary to what some people will say. And to understand why I picked Python as my goto language, you need to understand a little bit about the context of my decision.

Most of my career has been spent developing proprietary software for small groups of users (10-50) in the financial services industry, either for small departments in a larger firm or for all employees within a small company. There are a few important characteristics of this environment that are worth noting:

1. The financial services domain changes rapidly. New types of products are created every few months, new trading strategies are constantly developed, and the landscape for third-party products and financial market data evolves and changes at a very fast pace.
2. The number of users is relatively small (from a handful to dozens).
3. The users are not technical. They do not understand the complexities of software development and it can be very difficult to get the users to spend the time to define a problem with any level of clarity or detail that would provide a useful specification.
4. These are not technology companies. You will usually not find a lot of internal support to build a complete development team (e.g. with roles covering project management, quality assurance, tools developer, support, etc). If your boss doesn't see everyone on your team writing code all day then they will think something is wrong.

We can identify some requirements for a "good" programming language from the above points:

• The first point speaks to flexibility and the need to design for change. You want to develop solutions that can be easily modified as business needs change, and you want to choose tools that lend themselves nicely to this goal.
• The second point simply reminds us that we don't need to get carried away on complex architectures or get preoccupied with performance - simpler solutions will usually suffice.
• From point 3, I've learned that it really helps to know the financial business and to try and hire people that are conversant in the problem domain. But that's not easy so you can't rely on growing a team to meet the demands of the business. You may need get by with a smaller group and need to choose tools that really help your productivity.
• The last point has always been one of the harder things about working in finance - I usually don't have much in the way of technical peers to bounce ideas off of or to get a "second pair of eyes" when debugging. So its good to pick a language that has a rich community so you can fill that void outside of the office.

There are also a couple of important lessons I learned from previous jobs in this environment:

1. One language is better than many. I know people will often say you should use the right tool for the job, but in environments I've worked at where using multiple programming languages was the norm, its always been a disaster. There has always been way too much time spent on integration between the languages, and you will often end up with multiple implementations of the same components. Its also rare that every team member will be proficient in every language, so you don't have the ability to reassign people to different projects as easily.

2. Web applications work well for most problems. Web applications have a lot of benefits even for intranet applications. Deployment is dead simple and updates can be rolled out intraday. Users are familiar with the model and often don't require much training and support. Web applications perform well even for remote users connecting over a VPN. And modern development techniques have demonstrated that the web works nicely even for more dynamic, interactive types of applications.

The Language

Lessons Learned

I've used a number of programming languages over the years but have never really been passionate about any one. I started in C, enjoyed it a lot, and used it in the first few years of my career, and like many others, moved "up" to C++ for the next few years. But I never really liked C++. It always seemed that the benefits from the better OO support were outweighed by the added heft and complexity of the language. Add to that templates and MFC, and you sometimes feel like you are using a Rube Goldberg programming language.

From there I moved onto the more modern compiled languages, first Java and then later C#/.NET. While I found both those languages a breath of fresh air from the C++ world, I also found the simplicity of the languages lost in the sheer scale of the ecosystem they lived in. The class libraries were very large, but oddly void of some useful libraries (where is the FTP library in C#, for example?). And the index page of the Java docs for J2EE should borrow a line from Dante and warn the naive programmer "Abandon all hope, ye who enter here".

In the financial services area, there are also lots little data processing tasks that need to be performed (parsing broker files, web scraping, exporting data to vendors). I found C# too verbose for simple things like parsing a flat file and transforming the lines.

But there were a couple of things I did like about my experience using C#/.NET - it was the only language we used at that firm and the advantages of one language were real and notable:

• great reuse due to one common library
• good collaboration on the team as we all were "thinking in C#"
• a sort of network effect of shared knowledge that occurs when everyone is gaining expertise in the same set of tools

Interpreting the Options

With those experiences behind me, I started looking at interpretive programming languages. I had used Perl in the past and quickly ruled it out. I just think that it promotes "write only code" and that the language (and community) seems to value tricky one-liners more than explicit, understandable (and maintainable) code.

So I quickly whittled my choices down to Python and Ruby. After reading a lot of articles, primers and sample code, I found Python a better fit for my needs. I liked the focus on simplicity, explicitness, readability, practicality and "batteries included". It has great documentation and a very rich and helpful community. Ruby had a lot of the same things going for it but I found the libraries less comprehensive and developed and the community at least seemed smaller when I surfed around for various resources. (And frankly I don't find Ruby code that beautiful like your hear so often from many of its fans.)

The Framework

Once I settled on the language, I wanted to learn the "right" way to build web applications in Python. I had done a fair amount of ASP.NET in C# in my past job and found it lacking. If you stayed within the basics - using standard web controls and following the post-back model - it worked well. But for any real application, you quickly needed to go "outside the lines" where ASP.NET quickly became both complex and confining. Also, I just don't think that trying to make web programming like VB GUI programming is the right conceptual model for web apps.

In researching Ruby, I of course had run across Ruby on Rails. In some ways its probably more popular than Ruby itself. Python had a number of frameworks and tools to build web applications, but three stood out: Turbo Gears, Django and web.py. Turbo Gears and Django are higher level, full featured systems similar to Rails, while web.py is a much simpler and smaller toolkit, but quite compelling, nonetheless.

Since I was really looking for big productivity dividends, I ruled out web.py. There were too many things that the other frameworks had that I would need and I would end up having to build (or locate and integrate) on my own. While I liked both Turbo Gears and Django, I liked the consistency of Django that comes from a single complete project rather than a "best of breed" approach that Turbo Gears takes.

Results

I've now been using Python and Django for about 10 months and am very happy with my decision. I've written this blog and a simple todo app for the fun of it, and have built up about 12k lines of code on my work projects. At work I built up a nice collection of tools to help my dev process mostly by using standard Python packages. These include

• a web app for monitoring and managing scheduled tasks
• a unit testing framework that produces a complete web site of coverage charts, statistics and decorated source listings
• automated code documentation site

The more I learn about both Python and Django, the better I feel about the direction I've taken. Python seems to be a language that is simple when you are learning, but rich and powerful when you need it to be. Django shares these same qualities, and while still in its infancy, has already proven itself in many large scale web sites.

But probably the best thing I can say about Python is that I really enjoy using it. And that is "Why Python."

So now, a short two months later, I can safely say I'm done evaluating the Mac and now working on migrating to it completely. My MacBook has become my day-to-day personal machine and I really only use my existing Windows workstations at home to surf when I happen to be right in front of the machine.

I've been doing a test run with different types of data and applications to find the best way (for me) to use the Mac and to decide if there's any type of conversion I need to go through as I migrate my data. I ripped a handful of new music CDs and imported about 400 photos so I could play around with some multimedia content. I made a nice iPhoto book of my recent vacation in Vermont (way cool!), which has been a big hit with the grandparents. I checked out my subversion repository to try programming on the Mac, and outside of a few browser rendering quirks between IE and Safari, everything worked fine.

I had been using Microsoft Money for a long time to track my personal finances but was not really thrilled with the app (and the underwhelming annual upgrades). I did a lot of reading and reviewing of other products and finally settled on Moneydance. Migrating from my Money files wasn't that difficult - lots of people had done the same thing and posted notes to the discussion group - but I also wanted to clean up my data along the way and simplify my categorizations so there was a fair amount of work in getting it set up.

There was also one Windows-specific app that I'd become quite used to: KeePass. Its an application for managing the many passwords we all accumulate these days. It is an open-source application, but was developed for Windows so I wasn't quite sure how I was going to migrate the password database, plus I use it at both home and work so I wanted to find something that would work on OS X and Windows (so Keychain was out). It turns out I wasn't the only one with this problem as there is now a project for a Linux/OS X port of the application called KeePassX. Although its lacking some of the cooler features of the Windows version (like auto-type), it supports the same database format and that's what was important to me.

Backups

I've always been pretty good about backing up my data regularly and over the years had settled on scheduled tasks on all my machines that use Robocopy to mirror my data to my server, and then to mirror the server to a USB hard drive. I pretty quickly learned that rsync was the analogous technique for a *nix-based OS. A little googling led me to rsyncbackup which uses a Perl script to manage a set of sources, destinations and backupsets and invoke rsync to simplify the backup process. I configured it for a USB thumb drive, my home NAS and my externally-hosted Linux server.

Now that I have a feel for how I want to backup my data, I'll probably just write a Python script to drive the process. While rsyncbackup is pretty nice its a little more complicated than necessary (about half of the script is dedicated to parsing the config files, which I would just hard wire in as Python dicts). And its written in Perl: been there, done that, won't repeat.

With the recent demonstration of Panther (OS X 10.5) at the WWDC, I'm very curious to play around with Time Machine. I think that will augment my backup strategy, but not replace it. I think Time Machine will mostly eliminate the need to recover an accidentally deleted file from an external backup, but you'll still want to backup your current data regularly to some external devices - even time machines break once in a while.

Open Data Formats

As I've put more and more effort into building, organizing and cleaning my personal data library (music, photos, etc.), I've become more and more concerned about choosing an approach that gives me control over my data. Mark Pilgrim has written about this and for him the solution was to move away from the Mac. While I understand his line of reasoning, I'm not driven as much by the principles between open and proprietary file formats, but more by the pragmatism of being able to do what I want. (Or maybe I'm just one OS behind Mark?)

So I did a little research into Moneydance, iTunes and iPhoto (the most relevant applications for me right now, with respect to data ownership issues) and have found ways to preserve my data and meta data in ways that allow me to: 1) restore the information if data is lost or corrupted, and 2) export the data to other applications if I ever feel like I've outgrown any of the applications. Of course, this isn't as easy as it could be (i.e. simple import/export features built into each application). I'll have to resort to a little coding to make it work, but I'm fine with that.

Diving In

I'm now planning out how to migrate my other PCs to Macs. The only complication is that my wife is pretty set on Windows so I don't think I can cut her over (not without lots of tech support calls to my office). I'm going to first replace my other two desktop PCs at home with a 17" iMac and either another 20" iMac or a Mac Pro. To support my wife's need for Windows, I've worked out how to setup her Mac login such that it will automatically run Parallels in full screen when she logs in. To her it will look just like a Windows machine, but over time she'll have the opportunity to get more familiar with OS X without being forced to switch overnight.

The last bit of the puzzle is to get a Mac mini and use it as a home theater PC for viewing photos, listening to music and watching videos. It works pretty well out of the box with the Mac remote and Front Row, but there are other HTPC apps out there that are worth researching. There are also some cabling issues that I need to look into, but I've learned enough to know that it will work nicely. And it will look cool. (Why are all the Windows-based HTPCs big, slow and expensive?)

So after dipping my toes in the Apple Kool-aid water, I'm ready to dive right in...

I wanted to know the time spent generating the page in total as well as the breakdown between Python (Django) and the database. Additionally, I wanted to know how many times the database was queried as too many queries can be a hint that you aren't accessing the database efficiently.

Since I wanted the feature to work across any Django app, a custom middleware class seemed like the right approach. Writing a custom middleware is quite easy and, like so many other things in Django, well documented.

Calculating the Metrics

Luckily, the Django developers had already thought of the value of collecting some basic database statistics, so I simply had to figure out how to access them. When DEBUG mode is on (through your settings file), a database backend class wraps a cursor in a CursorDebugWrapper object which keeps a list of each SQL query executed along with its execution time. I just had to make sure DEBUG was enabled, if not already, and then add up the time of any calls incurred during this page's invocation.

The total time to process the view is measured in the middleware's process_view routine and then the time spent in Python (i.e Django) was easy to back out by timing the entire view call and then subtracting out the database time.

Viewing the Stats

The last part of the puzzle was how to put this information in the page itself. Normally in Django you would use a custom template tag for such a chore but I couldn't since I wouldn't have completed timing the view call until the template had already been rendered.

I decided to follow the spirit of template substitution by creating a special HTML comment that described the format of the output and then replacing the placeholder with the formatted output as the middleware returned the view to its caller.

The StatsMiddleware Class

The code is pretty straightforward. First, I save the state of the debug setting and then enable debugging. This triggers the use of the debug wrapper class in the database backend which will keep the stats of the database part of the execution. Then I save the size of the connection's queries where this debug info is stored so I will know which queries were additionally called during this view's invocation. Then I build the view, keeping track of its time

import re
from operator import add
from time import time
from django.db import connection

class StatsMiddleware(object):

    def process_view(self, request, view_func, view_args, view_kwargs):

        # turn on debugging in db backend to capture time
        from django.conf import settings
        debug = settings.DEBUG
        settings.DEBUG = True

        # get number of db queries before we do anything
        n = len(connection.queries)

        # time the view
        start = time()
        response = view_func(request, *view_args, **view_kwargs)
        totTime = time() - start

        # compute the db time for the queries just run
        queries = len(connection.queries) - n
        if queries:
            dbTime = reduce(add, [float(q['time']) 
                                  for q in connection.queries[n:]])
        else:
            dbTime = 0.0

        # and backout python time
        pyTime = totTime - dbTime

        # restore debugging setting
        settings.DEBUG = debug

        stats = {
            'totTime': totTime,
            'pyTime': pyTime,
            'dbTime': dbTime,
            'queries': queries,
            }

        # replace the comment if found            
        if response and response.content:
            s = response.content
            regexp = re.compile(r'(?P<cmt><!--\s*STATS:(?P<fmt>.*?)-->)')
            match = regexp.search(s)
            if match:
                s = s[:match.start('cmt')] + \
                    match.group('fmt') % stats + \
                    s[match.end('cmt'):]
                response.content = s

        return response

After the view is called, I restore the debug setting and save the stats in a dict which will be used for argument replacement during the output formatting step. As I said earlier, I use a sort of poor-man's templating to render the stats into the page on output. An HTML comment of the form

<!-- STATS: format_string -->

will be replaced with the output of format_string after formatting against the stats dict. This blog uses the string:

<!-- STATS: Total: %(totTime).2f Python: %(pyTime).2f
     DB: %(dbTime).2f Queries: %(queries)d -->

I put this inside the footer <div> and format through the CSS style sheet.

The replacement block in the code deserves a couple of comments. I wanted to make sure the string formatting was not applied against the entire content of the page so that random % symbols wouldn't cause problems. So I search for the special comment tag and then apply the formatting only to the subsection where it was found. The resulting content is then assembled with the untouched bounding parts of the original conent and the modified subsection.

Immediately Helpful

Soon after I got this working, it helped me identify a logic bug in my code. I had created a view that showed the position of an investment fund in a fairly large table (about 700 rows). The page took a few seconds to build and render. The statistics at the bottom of the page showed a whopping 2813 queries! Yikes!

The reason became immediately clear: each row in the table displayed a Position object as well as fields from the four other models it had ForeignKey relations with. Thus, each time a row was evaluated in the template, there were four individual database queries to look up the related data. I simply added select_related() to the end of my main query and the query count dropped to 4 and the execution time was cut to less than half the original 2.5 seconds. Nice.

Even though I'm mostly a Windows guy, I was intrigued by the Mac and impressed by what I'd seen from other Mac owners. The two factors that pushed me to side with the Mac were: 1) Windows compatibility through the Intel chip, and 2) changes in how I use a PC over time.

The Wintel compatibility with tools like Bootcamp and Parallels seemed like a nice insurance policy, if I really needed to do someting in Windows. But the second factor was really more important. I typically use my home PC for four things which all work fine on my Mac:

1. Surfing: most websites look and work fine in Safari, Firefox or IE. No bug surprise there.
2. Email: we use Exchange at work but there are a number of ways to access it from the Mac: the native Mail app can work with Exchange, you can use Microsoft's Entourage (essentially the Mac version of Outlook) or you can use Exchange's web interface - Outlook Web Access. I've tried them all so far and Entourage seems to be the nicest way to go.
3. Development: my development "stack" is now all open-source tools that are available on Windows and OS X as well as many other platforms like Linux. I use Apache, Python, MySQL and Django and have found no issues in running my apps on Windows, OS X or Linux for that matter (case in point: this site).
4. Media: or more specifically, organizing and working with photos, scans, music and video files. Since the data formats themselves are portable and well-supported, it comes down to which applications you prefer. This is a case where the Mac probably has more to offer and certainly is a strong selling point for the machine.

OS X

I really like OS X. it's intuitive, responsive, consistent and slick. Most people rave about the user interface - which is great - but I was really impressed with the integration between the modern GUI and the Unix layer beneath the covers. Other attempts I'd seen at this on Linux feel like a messy collection of glorified "etc" file editors, with no commonality or integration between them. In Linux I feel like the GUI just gets in the way and I often don't even bother installing X. But under OS X, the Terminal is really there as an extension when needed, which I'm finding I need less and less.

I haven't had my machine long enough to comment on stability. I assume it's very good but to be honest, I've had very good luck with Windows XP in that respect, too, so it's more the type of thing I've come to expect in a mature, modern operating system, rather than be surprised by.

Hooked?

it's too early to say if these are the first few steps towards a shift away from Windows in my home, or if I'll be content to just have the one-off Mac that I use only for myself. ( Or if you'll be seeing my Macbook on eBay once the honeymoon is over! But that seems doubtful at this point).

I have to admit that I simply like using my Mac - something I haven't really experienced since Windows 95 shipped. Sure, I've been happy when I've bought new PCs in the past, but even though the PC might be new, Windows is getting fairly old ... and just a little bit stale. At this point I'm not missing my old laptop or Windows for my home and casual PC use.

it's impressive how Apple consistently seems to outshine it's competitors in the areas of design and usability. The iPod is a great example, but you can find it in large, complex products like the Macbook and OS X as well as simpler products like their AC adapter or the iPod's lanyard headphones.

Apple must have a culture that values great design as it's so prevalent in their entire product line. Whatever it is, I'm glad they have it. And I'm glad I bought a Mac.

Update 1/14/2008: While this site is no longer built using Django, I thought I'd keep my Django-related posts up here as I've gotten a decent amount of feedback from people doing similar projects.

I've recently cutover from WordPress to a new blog which I developed using Django. I've been using Django at work for the past month and as I've gotten more familiar with it, I decided building a completely new blog would actually be fun and fairly simple. Besides, I've read about a few other people who have done the same thing and they say the hardest part was converting over their old blog entries. That should be cake for me as I only have a handful of posts to worry about!

If you haven't heard of it yet, Django is a framework for building web applications written in Python. It's an open-source project that spun out of an online news organization and thus has its roots in CMS applications. But it has now grown to support many other kinds of web applications. I stumbled onto Django when I was looking for a better way to build software after using C#/.NET and ASP.NET for the last two years. That led me to Python and I quickly found myself leaving C# behind with no regrets. Once I was sold on Python, I started looking at the "right" way to build web apps in Python. While there really isn't an answer to that question, Django comes pretty darn close.

Building the Blog

After reading the posts above about other Django-based blogs, I sat down to start my own. It took me about an hour to get a first cut of a working blog together, which I think is great, but some people seem fixated on the 15- or 20-minute project. I was basing some of my design on other people's older code base which changed considerably with the "magic-removal" relase so I had to convert things over (using the sophisticated engineering technigue of: refresh the browser, read the error message, fix the code, rinse, repeat).

I started with these fairly basic models for a Tag and a Post which gives you useful enough models for a surprsingly workable blog.

class Tag(models.Model):
    slug = models.SlugField(prepopulate_from=('title',), primary_key=True)
    title = models.CharField(maxlength=30)
    description = models.TextField(help_text='Short summary of this tag')

    def __str__(self):
        return self.title

    def get_absolute_url(self):
        return "/blog/tag/%s/" % self.slug

    class Admin:
        list_display = ('slug', 'title',)
        search_fields = ('title', 'description',)

    class Meta:
        ordering = ('title',)

A Tag is just really just a unique string used to categorize posts. In hindsight, I could have made this even simpler as I'm really just using a Tag's slug field. A Post is the model that really makes it a blog. Its also pretty simple, with the obvious fields: title, date and body, plus a the many-to-many tags relationship. It also has a slug field which is like a human-readable key to make it easier to refer to an object by a friendly name rather then a number.

class Post(models.Model):
    slug = models.SlugField(prepopulate_from=('title',), primary_key=True)
    tags = models.ManyToManyField(Tag)
    title = models.CharField(maxlength=80)
    date = models.DateTimeField(auto_now_add=True)
    body = models.TextField()

    def __str__(self):
        return self.title

    def get_absolute_url(self):
        return "/blog/%s/%s/" % (self.date.strftime("%Y/%m/%d").lower(),
                                 self.slug)

    class Admin:
        list_display = ('slug', 'title', 'date',)
        search_fields = ('title', 'body',)
        date_hierarchy = 'date'

    class Meta:
        get_latest_by = 'date'
        ordering = ('-date',)

Now that I'm a little more familiar with the problem, I could probably start from scratch and do it again in under 30 minutes, but why? In fact, I think someone should make a video of a real development session with all the typical mistakes and error messages, along with a voice over of how the user figured out the problem from the errors. Now that would be useful.

After getting the basic features done, I started working on the little tweaks that would make it more polished and easy to use. Some of these things came very easy but others were a little trickier. I'll post my comments on some of these features in the near futrue, as the information might be useful to others.

First I should start off with a bit of a disclaimer: this isn't the only question I ask as part of an interview process. On the contrary, I put candidates through a phone screen, have them take a written quiz (of which this is one of about 40 questions), and follow all that up with a take-home programming problem. No, the strcpy question is just one data point I use when evaluating programmers ... but it is probably one of the best.

To review, strcpy is the function for copying one string to another that is part of the standard library of the C programming language. Sounds simple, but asking someone to code it up is a bit of a loaded question, and that's what makes it so great for interviews. To implement the function, you have to know what it does and how strings are represented in C, which gets to one of the important design choices of C's creators: there is no native string type in C. Instead, a string is represented as a sequence of bytes (char values) in memory terminated with a byte containing zero. A pointer to the first byte in the sequence represents the string, but its really more of a convention then a first-class data type (although the compiler helps you a little by automatically appending a zero to string constants in memory).

Then there is the design of strcpy itself. It was designed to have the feel of an assignment statement for strings so where you might say

int a, b;
...
a = b;

to copy the value from b into a, you would likewise use

char *a, *b;
...
strcpy(a, b);

to copy the string value from b to a, that is, the destination is the first parameter (left-hand side) and the source is the second parameter (right-hand side), just like an assignment.

Another feature of C is that assignment statements themselves have a well-defined value and this is often used in shortcuts like

int val;
if ((val = getval()) != -1) {
    // do something with val
}

where we can call a function that returns a value, assign it to a variable and test its value in one expression. A (little-known) feature of strcpy is that it returns the value of the destination pointer as a convenience to the caller so you can copy a string, and use the value in one expression as in

char *a, *b;
...
printf("%s\n", strcpy(a, b));

Admittedly, not a feature I've used much, but experienced C programmers are usually aware its there.

Another interesting thing about implementing strcpy is that the solution is introduced, discussed and refined in the C "bible": The C Programming Language by Brian Kernighan and Dennis Ritchie, more commonly referred to as the "K&R book." Anyone who really wanted to learn C would have at least heard of this book and hopefully been curious enough to read it. If they had (and they paid attention), then they would already know the answer. Experience has shown me, though, that it doesn't appear to be as popular as I would have thought - at least not with the candidates that have crossed my path.

Phrasing the Question

The strcpy question was one of a number of problems I put on a written quiz for interviewing programmers. I liked having candidates write it on paper because it avoided the problem of working on a computer where different programmers use different tools - everyone should be equally familiar with paper and pencil. It also avoided the distractions of a development environment where the person might be messing around with build settings and key bindings rather than focusing on the problem at hand.

As for how to phrase the question, initially I was wary of giving away too much with the signature of the function, and part of the reason for asking this specific question was to see if users were familiar with such a ubiquitous function, so my first version was simply

Implement the C standard library function: strcpy

I figured most people would know what it was and what it did so there wasn't really a need to be more explicit. And if they didn't know the function, then that told me a lot - maybe more than whether or not they could code it. You have to understand, the interviewee had already done well enough on a phone screen to be invited in for a round of interviews. I figured wrong. Asking the question in this way led to very few correct answers. Some people had not heard of the function, many people who knew what it did got the source and destination arguments mixed up and practically no one got the return value right. So for the second version I gave up on people knowing the correct signature from memory and rephrased the question to be more explicit (and more in line with ANSI C):

Implement the following C library function:

char* strcpy(char* s1, const char* s2) {
    ...
}

Note that I didn't give away the source and destination arguments by name but I thought most people would see the const modifier hint. I also expected the return type of the function to jog a few people's memories. It turns out that the const hint was not strong enough, however, as many people continued to confuse the source and destination arguments. A few people understood that they should be returning something, but even with the prototype, very few people actually got that part right.

I tweaked the question again to arrive at the third version. I simply gave up on the return type in order to simplify the problem so they could focus on the basic algorithm. I also named the arguments hoping to clear up the confusion:

Implement the following C library function:

void strcpy(char* dst, const char* src) {
    ...
}

So now I was leading them to the water, and was expecting many correct, or nearly correct solutions. Did I ever say I was a bit of an optimist? How foolish I was to think my new version of the question was clear enough. Granted, most people started off on the right track copying chars from the const src pointer to the dst pointer. (No one ever asked why its return type was not char * so I felt validated on that omission.) But wait, you can do it without calling strlen - think about it! And why are you calling malloc? Or memcpy? Okay, one more refinement and I think I've got it:

Implement the following C library function, without calling any other routines:

void strcpy(char* dst, const char* src) {
    ...
}

Whew!

Evaluating the Answer

There were five key things I looked for when evaluating an answer.

1. Did they know what the function was supposed to do? That is, was their algorithm an attempt at an actual copy of a string?
2. Was the algorithm correct? Did they copy all the characters? Did they copy the terminating null? (Later, when we went through the quiz, I'd ask the candidate to walk through their code for the string "Hi" to test/break it.)
3. Did they follow my instructions - no system calls or other routines?
4. Did they get the src and dst arguments right?
5. Did they use arrays or pointers? This was more to see how comfortable they were with pointers.

I also considered a few other points for "extra credit": Did they know about the return value? Did they write defensively (check for null arguments)? As you can see, there is a lot to be learned from such a basic question. I also believe that people are creatures of habit: they tend to do things the same way in this setting as they might on the job. If they were sloppy and careless on the interview ... well caveat emptor.

The other thing I was always thinking about was the context they were in. Sure, interviewing can be stressful, and there were many other problems on the quiz so they couldn't spend all their time on this one, but this was an interview for a job! I expected people to be going a little over the top in order to give a good impression, so details were important. And while I wouldn't say that any one thing on an interview should be a sole determination if someone is hired or not, it was hard for me to get excited about hiring someone that couldn't get this right.

The Solution

Although there are many ways to solve this problem, we can look to the K&R book for the classic solution.

void strcpy(char* dst, const char* src)
{
    while (*dst++ = *src++)
       ;
}

Solution 1: The "classic" K&R answer

Its simple, concise and correct. We can quibble about where to put the semicolon or if we should use braces, but that is all just style - the statement that does all the work is in the test of the while. This solution uses a number of features of the C language to get a lot done in essentially one line of code:

• A char is copied from one pointer to another using the dereferencing ("*") operator
• Post-incrementing is used to march the pointers forward in memory after the char is copied
• It takes advantage of the fact that the result of an assignment is the value of the statement of itself
• The terminating zero of a C-string is used as a boolean test for the while loop

That's a lot packed into one simple expression! These are all features of C that its creators designed into the language for a purpose. This expression, and slight variations of it, are so common that they are idioms in C programming and any C programmer will likely come across code with statements like these time after time. They better fully understand what's going on here if they are going to master the language (or at least to read and understand other people's code).

I mentioned earlier that the "real" version of strcpy returns a copy of the destination pointer as a convenience. Thus, a more correct version of the function would be

char* strcpy(char* dst, const char* src)
{
    char* ret = dst;

    while (*dst++ = *src++)
        ;

    return ret;
}

Solution 2: Handle the return value correctly

That is, we save a copy of the destination pointer and return it after we've copied the string. While that is more correct, it doesn't demonstrate any real coding skills - its really more of a trivia question about strcpy. That's why I eventually rephrased my interview question to use a void return type (and of course because no one was getting that detail correct, anyway).

There are a couple of common variations on this solution: using a different looping construct, and using arrays rather than pointers. The following solution shows one variant using a for loop.

char* strcpy(char* dst, const char* src)
{
    int i;
    for(i = 0; src[i] != 0; i++) {
        dst[i] = src[i];
    }
    dst[i] = 0;

    return dst;
}

Solution 3: Using array notation

Since we never modify dst - we use i as an index to offset from the starting address - there is no reason to make a temporary copy for the return value. But since we aren't copying and testing for null in the same statement here, the terminating zero is not copied in the loop. Thus we need to terminate the destination string outside of the loop.

These are all correct solutions in that they copy the string including the terminating null. And although I personally prefer a version using pointers as it demonstrates a degree of understanding and confidence in using them, the array version is perfectly fine.

There are, of course, dozens of other variations that could be written. We could use a for with pointers, use a while loop with arrays, use a do loop - we could even use recursion if we wanted to be silly (and inefficient). But most often I've found good solutions will look a lot like one of the snippets above.

Answers

I've taken a not-so-random sample of some past solutions to show some of the classic mistakes people make. For dramatic effect, I'll start with the good ones and work my way down through the common mistakes people make, all the way to the really awful answers that I'm not even sure what they were trying to do. Remember, these are actual answers from actual programmers. Additionally, these were people who were often gainfully employed at the time of the interview, had been recommended by recruiters or other people, and had already been screened through a phone interview. All of these snippets are exactly as answered including syntax errors, comments, creative operators, etc.

void strcpy(char* dst, const char* src)
{
    for (; dst != Null && src != Null;)
        *dst++ = *src++;
}

Answer 1

So close! If they would have just replaced the test clause of the for loop with its body, they would have nailed it. But they got confused.

• They assumed "Null" is some pre-defined constant for zero
• They compared addresses and not the value pointed to by the address so this would not terminate
• Even if they got the loop test correct, they wouldn't have copied the terminating null character

But all in all not a bad answer and the use of pointers for copy-and-increment is promising. You can tell that they've either read up on C or used it in the past, but probably aren't using it day to day.

void strcpy(char* dst, const char* src)
{
    if (src == NULL) return;
    if (dst == NULL) return;

    while (src[i++] != '\0') {
        dst[i] = src[i];
    }
    dst[i] = '\0';
}

Answer 2

Hey, defensive programming! I like that, and I see him copying the null at the end, could we have a winner? Oh, I'm sorry, you got the last char but forgot the first one (and you forgot to define i).

void strcpy(char* dst, const char* src)
{
    while(*src)
        *dst++ = *src++;
}

Answer 3

Another close one! They seem comfortable with pointers and their use in the copy-and-increment idiom. But the while test is going to kick you out of the loop before you copy the terminating 0 (which you would have noticed if you tested it). Sorry.

void strcpy(char* dst, const char* src)
{
    *dst = *src;
}

Answer 4

Well, it does work for the special case of strcpy(buf, ""), and I have to admit, it is fast, but I was looking for something a little more ... correct.

void strcpy(char* dst, const char* src)
{
    int counter = 0;
    while (*(src + counter) <> '/0') {
        *(dst + counter) = *(src + counter);
        counter++;
    }
}

Answer 5

Well the syntax for "not equals" smells a little like VB and there seems to be confusion on the syntax for an escape character. Overlooking those points, there is still the problem with the terminating null, and while the pointer expressions are valid, I find them less clear than the examples that increment the pointers.

At least in answers 1-5 you can see some familiarity with C and the use of pointers, plus its clear that people knew what strcpy did, if not exactly how it did it. Now let's move on to some more "creative" answers.

void strcpy(char* dst, const char* src)
{
    char i = null;
    do while i <> '\n'
    {
        i = &src;
        &dst = i;

        dst++;
        src++;
    }
}

Answer 6

There are a number of issues with this answer:

• Confusion between the dereferencing and address operators (*/&)
• Wrong syntax for a do/while loop
• Wrong syntax for "not equals"
• They are looking for a terminating end of line character, not a null
• It doesn't copy the terminating null (or \n for that matter)

void strcpy(char* dst, const char* src)
{
    int idst, isrc;
    idst = 0; isrc = 0;
    while (src[isrc] != "\0")
        dst[idst++] = src[isrc++];
}

Answer 7

This time they confused the syntax of a char constant with a string constant. This is actually a very nasty bug (if the compiler didn't warn on type mismatches) as "0" evaluates to an address that stores a string constant so the while would never terminate. Oh, and they also forgot the null (are you beginning to see a theme?).

void strcpy(char* dst, const char* src)
{
    int ilen = 0;
    char *trav, *trav2;

    for(trav = src; *trav++; ilen++);
    free dst;
    dst = malloc(ilen + 1);
    trav2 = dst;
    for (trav = src; *trav2 = *trav++; trav2++);
}

Answer 8

Where do I start?

• The first for is counting the length of src (we'll see why in a couple of lines)
• Then they try to free dst (with a syntax error on the free call). That's going to surprise the caller!
• Now they get (leak) a new chunk of memory that the caller will never know about
• The second for then actually copies the string correctly! Yeah!

What's funny is if they would have deleted the first three lines (the for, free and malloc), this would be correct (although a bit overly confusing with inconsistent initialization and incrementing statements). Because those three lines are there, however, it copies the string to our newly allocated chunk of memory that no one will know about as soon as the function exits. But I'm being pedantic - it may never get that far since the free call would likely cause a crash. There's also the issue of not following instructions with the use of malloc (at least they didn't call strlen, though).

The use of malloc and free is particularly troubling. This demonstrates that the candidate misunderstands a couple of the basic design decisions that went into the C language: that it has no built-in memory operators (i.e. no new or delete), and that the developer is completely responsible for memory management. Allocating memory inside a function that advertises only to copy a string would violate this rule. More importantly, strcpy requires the user pass in the destination address, thus it should be clear that the user is responsible for making sure that address points to a valid destination with enough space to hold the string.

Note: The strdup function, on the other hand, does violate this rule but clearly states this in its documentation. You can also infer this from its design. It takes only one argument - the source string - and returns the address of a new duplicate of the string. Since the user cannot pass the destination address, the function must allocate it for them - and return it. It is essentially a memory allocator that fills the new chunk of memory for the user, and I think would have been more appropriately called salloc and put in the malloc.h include file rather than string.h. Although strdup may be convenient in certain cases, I still prefer not to use it. I prefer controlling how the memory for the copy is obtained - I might have a static char array that I reuse, or an array on the stack. I think it is much more clear to take responsibility for obtaining the block of memory and calling strcpy with a pointer to it, rather than calling strdup and having to remember to match up a free with each call to it - its difficult enough matching malloc and free calls.

void strcpy(char* dst, const char* src)
{
    int size = strlen(src);
    malloc(dst, size)  // syntax?

    for (int i = 0; i < size; i++) {
        dst[i] = src[i];
    }
    return dst;
}

Answer 9

I'm disappointed from the first line. They didn't follow instructions and made a call to strlen. And things only get worse with the call to malloc. But they returned dst! Nice! (Except of course its pointing to a non-terminated string and a different address then was passed in - assuming they would have eventually fixed the malloc call to: dst = malloc(size);.

void strcpy(char* dst, const char* src)
{
    free(dst);
    dst = malloc(sizeof(src));
    if (dst) {

    } else { // error; }
}

Answer 10

Let's see - system calls, freeing dst, and hopefully the string is 3 bytes (4 with the terminating zero) since they are allocating a chunk of memory for the size of an address, and not the length of the string it points to. But then the solution fizzles out. Come on, you weren't shy about using system calls. Why not round it out with a call to memcpy to at least finish it!

Looking Back

There are a number of things I learned about programmers from this one simple question. You can argue whether the question is a good one, whether my criteria for evaluating answers was fair, and whether this question is even relevent anymore. But one thing that you can't argue with is the value of asking people to demonstrate their knowledge by writing code, plus the comparitive value in asking the same question over and over. You learn a lot about people by asking them to solve something real that they should be familiar with and should understand.

First of all you learn how wildly one programmer's knowledge about a language can vary from another's. When someone says they "know" a language, what does that really mean? Does that mean they know the syntax? That they are comfortable with the standard library? That they understand the internal representation of data structures? That they are comfortable with pointers?

You also find out if people are curious or not. When they were learning C, did they find out about the K&R book and read it on their own? Sometimes if a candidate does well I'll have them come back for a second round of interviews and I might ask them, how would you do it now? I love it when they ran home, read up on C and learned how to do it better. Or they proactively just emailed me an improved version because they couldn't let go of it.

What about their style? Is the code legible? Organized? Were they careful? Did they bother to test with the most trivial example to see if it works? Granted, strcpy is too small to answer some of these questions but it wasn't the only programming problem they were given. The point is that looking at someone's code tells a lot about them - it is what you are hiring them to produce and you want to know how they go about creating it. Would you hire a photographer without looking at their portfolio?

Some people may find it disagreeable that I give someone a written quiz as it creates a stressful situation. I agree. I've always worked in a high pressure environment and want to make sure the candidates can handle that pressure (most of my career I worked on trading desks of Wall Street firms).

I've done enough of this to know that this isn't a statistical anomoly. The truth is that there are a lot of bad programmers out there. That's not to say that this simple question is the one true determinator of a good or bad programmer - its just one of many data points I've used in interviewing programmers. But I'd certainly make the argument that its positively correlated with a person's skills and knowledge and all other factors being equal, I'd much prefer hiring someone who nailed it then someone who took twenty lines to get it wrong.