Advanced programming tips, tricks and hacks for Mac development in C/Objective-C and Cocoa.

5 key-value coding approaches in Cocoa

Key-value coding (KVC) is a way of decoupling a generic action from the specific properties it may need to act upon. It is most commonly associated with the NSKeyValueCoding protocol but there are a number of other ways to achieve the same effect. In this post, I look at why key-value coding is important and show you 5 different ways — each with their own particular advantages — to implement this pattern.

Introduction

The NSKeyValueCoding protocol has been in Cocoa since Mac OS X 10.0 but it first made a real mark in Mac OS X 10.3 when user-interface bindings demonstrated their potential — allowing user-interface controls to connect to properties on objects by configuring the data, not changing the code.

While the core concepts in key-value coding exist (or can be implemented) in most programming frameworks, key-value coding is both prevalent and pervasive throughout Cocoa in a way that is uncommon, certainly among other compiled application frameworks.

Despite its prevalence, you can avoid key-value coding if you choose. However, Apple introduced it for a reason — it is a simple design pattern that can be highly beneficial to your code. To that end, I'm going to spend this post talking about what key-value coding is, how it can improve your code and different approaches you can use to achieve the effect.

What is key-value coding?

The idea behind key-value coding is pretty simple: instead of directly getting and setting specific properties on an object, key-value coding involves passing in a "key" (usually a string) and getting or setting the property associated with that key.

This may sound a lot like NSDictionary

For example:

// Set a property directly...
someObject.someProperty = someValue;

// ...or set the same property using key-value coding
[someObject setValue:someValue forKey:@"someProperty"];

Why would you want to do this? The answer is that it decouples the action of setting a value for a property from the specific property involved.

Imagine a table for editing names and addresses:

tablescreenshot.png

Without key-value coding, the NSTableViewDataSource method to handle an edit for one of the rows might look like this:

- (void)tableView:(NSTableView *)aTableView
    setObjectValue:(NSString *)anObject
    forTableColumn:(NSTableColumn *)aTableColumn
    row:(int)rowIndex
{
    if ([[aTableColumn identifier] isEqual:@"name"])
    {
        [[records objectAtIndex:rowIndex] setName:anObject];
    }
    else if ([[aTableColumn identifier] isEqual:@"address"])
    {
        [[records objectAtIndex:rowIndex] setAddress:anObject];
    }
}

With key-value coding, the method becomes:

- (void)tableView:(NSTableView *)aTableView
    setObjectValue:(NSString *)anObject
    forTableColumn:(NSTableColumn *)aTableColumn
    row:(int)rowIndex
{
    [[records objectAtIndex:rowIndex] setValue:anObject forKey:[aTableColumn identifier]];
}

I spoke about bindings before: this approach isn't using user-interface bindings (that would require no code at all). This is really showing a simplified way that bindings might be implemented.

This key-value coding approach is better because it doesn't have to handle the edit of each property as a separate condition. This is the essence of key-value coding.

And key-value coding remains just as efficient as the data set increases in size. A table with 1,000 columns would require the same amount of code to edit.

KVC approach 1: NSKeyValueCoding protocol

All the key-value coding that I've show so far uses the NSKeyValueCoding protocol. Actually, I call it a protocol but it is actually an "informal protocol" (a category on NSObject).

This category implements the setValue:forKey: and valueForKey: methods that you can use for setting and getting values by NSString keys.

Advantages
  • Finds getter and setter methods automatically and will even get or set ivars directly if a getter or setter can't be found. This means that most properties automatically support NSKeyValueCoding. Learn more about the search paths in my Key Value Information post.
  • Includes key paths (for traversing multiple properties).
  • Integrates with NSKeyValueObserving for implementing the Observer design pattern.
  • Offers fallbacks and ways of dealing with undefined keys.
Disadvantages
  • The extended search path makes this one of the slowest key-value coding approaches (see my earlier performance-related post on Replacing Core Data Key Paths).
  • Requires either a method or ivar on the class matching the property name that can be found by NSKeyValueCoding.
  • Only supports NSStrings as property keys.

KVC approach 2: Manual subsets of NSKeyValueCoding behavior

The NSKeyValueCoding protocol looks up methods by selector names and looks up ivars by name.

This is work that you can do yourself.

// Manual KVC setter method implementation
NSString *setterString = [@"set" stringByAppendingString:[someKeyString capitalizedString]];
[someObject performSelector:NSSelectorFromString(setterString) withObject:someValue];

// Manual KVC ivar setter
object_setInstanceVariable(someObject, someKeyString, someValue);

Why would you do this instead of using the NSKeyValueCoding approach? You would only use this in situations where you want to avoid methods or ivars that would ordinarily be found by NSKeyValueCoding. This approach allows you to define your own lookup path.

Advantages
  • More control over the lookup path than with NSKeyValueCoding.
  • Potentially faster than NSKeyValueCoding.
  • Will work for classes that don't inherit from NSObject and hence don't have NSKeyValueCoding implementations.
  • Manual method implementations can get and set non-object values.
Disadvantages
  • Less flexible than NSKeyValueCoding.
  • In most cases, it is more work than using NSKeyValueCoding.

KVC approach 3: Associated objects

The Objective-C 2.0 runtime (used on the iPhone and 64-bit Mac OS X apps) allows you to set any object to be associated with any other object. This allows any object in the runtime to have an arbitrary set of extra properties set by key, without support from ivars or methods on the object itself.

objc_setAssociatedObject(someObject, someKey, someValue, OBJC_ASSOCIATION_RETAIN_NONATOMIC);

The main reason why you would use this approach is that you want to set properties on an object from the outside — i.e. without the object supporting, being involved with or even knowing about the property access. Properties can be set on an object by other parts of the program for their own purposes.

Advantages
  • No support from the object (methods or ivars) required.
  • Key can be any pointer (so can the object if OBJC_ASSOCIATION_ASSIGN is used).
  • Potentially the fastest KVC approach.
Disadvantages
  • Key is a pointer, not an object, so if an object is used, it must be a pointer-unique object (i.e. won't work if you try to get and set with different allocations of the same string value).
  • Does not affect ivars or methods on the object. Generally this means if you want the object itself to know about the change, you must use a different approach.

KVC approach 4: selectors as keys

Key-value coding is primarily about looking up a property for a key and then acting upon the property found during lookup.

Objective-C has a lookup at its very core — the method lookup. The keys for this lookup are selectors.

objc_msgSend(someObject, someSetterSelector, someValue);

This approach is similar to manually implementing the method part of NSKeyValueCoding but rather than forming a selector string from the key and then looking up the selector string, this approach opts to use the selector as the key.

The disadvantage this approach has is that separate selectors are needed for getting and setting.

Advantages
  • Fastest approach that goes through methods (which is good since methods are overrideable and hence more subclass friendly).
  • Can get and set non-object data (although objc_msgSend_fpret and objc_msgSend_stret need to be used to get float, double and struct properties).
Disadvantages
  • Different keys required for getting and setting.
  • Selectors are not objects and are therefore can't be stored directly in Objective-C arrays and dictionaries (must use CoreFoundation or NSValue wrappers).

KVC approach 5: do it yourself

The final approach to key-value coding is to handle the implementation yourself. This is something you would do if you needed maximum flexibility (for handling unusual keys/values) or wanted to expose different key-value sets from a single object.

The easiest way to do this is to expose a getter and a setter method and simply get or set the values on a dictionary contained by the object.

- (void)setCollectionValue:(id)value forKey:(NSString *)key
{
    [collectionDictionary setObject:value forKey:key];
}

- (id)getCollectionValueForKey:(NSString *)key
{
    return [collectionDictionary objectForKey:key];
}

To handle the internal storage of the values, you could use any of the key-value storage structures in Cocoa:

  • NSMutableDictionary
  • NSMapTable
  • CFMutableDictionaryRef
  • associated objects on self or other objects (see above)

or your own storage solution.

Advantages
  • A single object can exposes multiple, separate collections.
  • Can get and set any data type supported by the underlying collection.
  • The most flexible approach for fallbacks and handling special cases.
Disadvantages
  • Must be implemented by the target class (will not work for arbitrary objects).
  • Doesn't interoperate with NSKeyValueObserving or any of the other NSKeyValueCoding concepts.

Conclusion

Using key-value coding is not mandatory — it is certainly possible to implement whole projects without it. However, it is one of the best code patterns for reducing repetitious code and making classes more reusable by decoupling actions from properties and data.

As you can see, there are a number of different kinds of key-value coding you can use in your program. NSKeyValueCoding is probably the most flexible, reusable and best supported throughout Cocoa, so unless you need one of the advantages of the other approaches or you want to keep your solution to a problem narrow, it is probably the best approach to use.

Read more...

What is a meta-class in Objective-C?

In this post, I look at one of the stranger concepts in Objective-C — the meta-class. Every class in Objective-C has its own associated meta-class but since you rarely ever use a meta-class directly, they can remain enigmatic. I'll start by looking at how to create a class at runtime. By examining the "class pair" that this creates, I'll explain what the meta-class is and also cover the more general topic of what it means for data to be an object or a class in Objective-C.

Creating a class at runtime

The following code creates a new subclass of NSError at runtime and adds one method to it:

Class newClass =
    objc_allocateClassPair([NSError class], "RuntimeErrorSubclass", 0);
class_addMethod(newClass, @selector(report), (IMP)ReportFunction, "v@:");
objc_registerClassPair(newClass);

The method added uses the function named ReportFunction as its implementation, which is defined as follows:

void ReportFunction(id self, SEL _cmd)
{
    NSLog(@"This object is %p.", self);
    NSLog(@"Class is %@, and super is %@.", [self class], [self superclass]);
    
    Class currentClass = [self class];
    for (int i = 1; i < 5; i++)
    {
        NSLog(@"Following the isa pointer %d times gives %p", i, currentClass);
        currentClass = object_getClass(currentClass);
    }

    NSLog(@"NSObject's class is %p", [NSObject class]);
    NSLog(@"NSObject's meta class is %p", object_getClass([NSObject class]));
}

On the surface, this is all pretty simple. Creating a class at runtime is just three easy steps:

  1. Allocate storage for the "class pair" (using objc_allocateClassPair).
  2. Add methods and ivars to the class as needed (I've added one method using class_addMethod).
  3. Register the class so that it can be used (using objc_registerClassPair).

However, the immediate question is: what is a "class pair"? The function objc_allocateClassPair only returns one value: the class. Where is the other half of the pair?

I'm sure you've guessed that the other half of the pair is the meta-class (it's the title of this post) but to explain what that is and why you need it, I'm going to give some background on objects and classes in Objective-C.

What is needed for a data structure to be an object?

Every object has a class. This is a fundamental object-oriented concept but in Objective-C, it is also a fundamental part of the data. Any data structure which has a pointer to a class in the right location can be treated as an object.

In Objective-C, an object's class is determined by its isa pointer. The isa pointer points to the object's Class.

In fact, the basic definition of an object in Objective-C looks like this:

typedef struct objc_object {
    Class isa;
} *id;

What this says is: any structure which starts with a pointer to a Class structure can be treated as an objc_object.

The most important feature of objects in Objective-C is that you can send messages to them:

[@"stringValue"
    writeToFile:@"/file.txt" atomically:YES encoding:NSUTF8StringEncoding error:NULL];

This works because when you send a message to an Objective-C object (like the NSCFString here), the runtime follows object's isa pointer to get to the object's Class (the NSCFString class in this case). The Class then contains a list of the Methods which apply to all objects of that Class and a pointer to the superclass to look up inherited methods. The runtime looks through the list of Methods on the Class and superclasses to find one that matches the message selector (in the above case, writeToFile:atomically:encoding:error on NSString). The runtime then invokes the function (IMP) for that method.

The important point is that the Class defines the messages that you can send to an object.

What is a meta-class?

Now, as you probably already know, a Class in Objective-C is also an object. This means that you can send messages to a Class.

NSStringEncoding defaultStringEncoding = [NSString defaultStringEncoding];

In this case, defaultStringEncoding is sent to the NSString class.

This works because every Class in Objective-C is an object itself. This means that the Class structure must start with an isa pointer so that it is binary compatible with the objc_object structure I showed above and the next field in the structure must be a pointer to the superclass (or nil for base classes).

As I showed last week, there are a couple different ways that a Class can be defined, depending on the version of the runtime you are running, but yes, they all start with an isa field followed by a superclass field.

typedef struct objc_class *Class;
struct objc_class {
    Class isa;
    Class super_class;
    /* followed by runtime specific details... */
};

However, in order to let us invoke a method on a Class, the isa pointer of the Class must itself point to a Class structure and that Class structure must contain the list of Methods that we can invoke on the Class.

This leads to the definition of a meta-class: the meta-class is the class for a Class object.

Simply put:

  • When you send a message to an object, that message is looked up in the method list on the object's class.
  • When you send a message to a class, that message is looked up in the method list on the class' meta-class.

The meta-class is essential because it stores the class methods for a Class. There must be a unique meta-class for every Class because every Class has a potentially unique list of class methods.

What is the class of the meta-class?

The meta-class, like the Class before it, is also an object. This means that you can invoke methods on it too. Naturally, this means that it must also have a class.

All meta-classes use the base class' meta-class (the meta-class of the top Class in their inheritance hierarchy) as their class. This means that for all classes that descend from NSObject (most classes), the meta-class has the NSObject meta-class as its class.

Following the rule that all meta-classes use the base class' meta-class as their class, any base meta-classes will be its own class (their isa pointer points to themselves). This means that the isa pointer on the NSObject meta-class points to itself (it is an instance of itself).

Inheritance for classes and meta-classes

In the same way that the Class points to the superclass with its super_class pointer, the meta-class points to the meta-class of the Class' super_class using its own super_class pointer.

As a further quirk, the base class' meta-class sets its super_class to the base class itself.

The result of this inheritance hierarchy is that all instances, classes and meta-classes in the hierarchy inherit from the hierarchy's base class.

For all instances, classes and meta-classes in the NSObject hierarchy, this means that all NSObject instance methods are valid. For the classes and meta-classes, all NSObject class methods are also valid.

All this is pretty confusing in text. Greg Parker has put together an excellent diagram of instances, classes, meta-classes and their super classes and how they all fit together.

Experimental confirmation of this

To confirm all of this, let's look at the output of the ReportFunction I gave at the start of this post. The purpose of this function is to follow the isa pointers and log what it finds.

To run the ReportFunction, we need to create an instance of the dynamically created class and invoke the report method on it.

id instanceOfNewClass =
    [[newClass alloc] initWithDomain:@"someDomain" code:0 userInfo:nil];
[instanceOfNewClass performSelector:@selector(report)];
[instanceOfNewClass release];

Since there is no declaration of the report method, I invoke it using performSelector: so the compiler doesn't give a warning.

The ReportFunction will now traverse through the isa pointers and tell us what objects are used as the class, meta-class and class of the meta-class.

Getting the class of an object: the ReportFunction uses object_getClass to follow the isa pointers because the isa pointer is a protected member of the class (you can't directly access other object's isa pointers). The ReportFunction does not use the class method to do this because invoking the class method on a Class object does not return the meta-class, it instead returns the Class again (so [NSString class] will return the NSString class instead of the NSString meta-class).

This is the output (minus NSLog prefixes) when the program runs:

This object is 0x10010c810.
Class is RuntimeErrorSubclass, and super is NSError.
Following the isa pointer 1 times gives 0x10010c600
Following the isa pointer 2 times gives 0x10010c630
Following the isa pointer 3 times gives 0x7fff71038480
Following the isa pointer 4 times gives 0x7fff71038480
NSObject's class is 0x7fff710384a8
NSObject's meta class is 0x7fff71038480

Looking at the addresses reached by following the isa value repeatedly:

  • the object is address 0x10010c810.
  • the class is address 0x10010c600.
  • the meta-class is address 0x10010c630.
  • the meta-class's class (i.e. the NSObject meta-class) is address 0x7fff71038480.
  • the NSObject meta-class' class is itself.

The value of the addresses is not really important except that it shows the progress from class to meta-class to NSObject meta-class as discussed.

Conclusion

The meta-class is the class for a Class object. Every Class has its own unique meta-class (since every Class can have its own unique list of methods). This means that all Class objects are not themselves all of the same class.

The meta-class will always ensure that the Class object has all the instance and class methods of the base class in the hierarchy, plus all of the class methods in-between. For classes descended from NSObject, this means that all the NSObject instance and protocol methods are defined for all Class (and meta-class) objects.

All meta-classes themselves use the base class' meta-class (NSObject meta-class for NSObject hierarchy classes) as their class, including the base level meta-class which is the only self-defining class in the runtime.

Read more...

Getting the subclasses of an Objective-C class

Getting the full list of subclasses for a class is a fairly simple task but it requires some uncommon runtime functions which can make the process more difficult. In this post, I look at how a Class is defined in Objective-C and two completely different ways of working out the subclasses.

Introduction

Given the high level of reflection and introspection in the Objective-C runtime, you might expect that you would simply find a class_getSubclasses(Class parentClass) in the Objective-C Runtime API but no such function exists.

There are a few possible reasons why this function might have been omitted — dynamic class creation and loading, threading and locking concerns, historical omissions in the class_t structure, premature optimization considerations or even an attempt to deliberately guide programmers away from certain designs — but the result is that you must work out the subclasses for yourself.

Note: fetching the subclasses of a class is an unusual thing to do and is usually done as an after-the-fact change to an API you don't control or as a debug-only introspection or hack. Generally, designs where subclasses explicitly register themselves work better than trying to find subclasses at runtime. For example, in my Simple, Extensible HTTP Server, every HTTPResponseHandler must include:

+ (void)load
{
    [HTTPResponseHandler registerHandler:self];
}

which allows subclasses of HTTPResponseHandler that "opt-in" to be discovered by the parent when it is looking for a handler to handle a request.

Fetching subclasses is then used in situations where you can't include a load method like this or you can justify not wanting to do this (e.g. debug-only verification code which checks that all HTTPResponseHandlers you've written actually do register themselves).

Filter the list of all classes

The traditional definition of a class in Objective-C looks like this:

struct objc_class {
    Class isa;
    Class super_class;
    const char *name;
    long version;
    long info;
    long instance_size;
    struct objc_ivar_list *ivars;
    struct objc_method_list **methodLists;
    struct objc_cache *cache;
    struct objc_protocol_list *protocols;
};

Classes contain pointers to their superclass but contain no pointer to subclasses.

Since there is no specific function to access the subclasses, the only approach using public APIs is to get the list of all classes in the runtime and test each one to see if it is a subclass of the class in question.

If this sounds like a heavy-handed approach, you're probably right. In Mac OS X 10.6.2, there are 527 classes in Foundation alone and 1966 in the Cocoa framework. This doesn't include all the classes your project will add, plus classes in other frameworks. Of course, interrogating all of these only takes a millisecond or two but it's still not something you should do in a tight loop — if you need to repeatedly use the list of subclasses for a class, you'll want to cache it somewhere.

Fetching all the classes in the runtime is pretty simple:

int numClasses = objc_getClassList(NULL, 0);
Class *classes = NULL;

classes = malloc(sizeof(Class) * numClasses);
numClasses = objc_getClassList(classes, numClasses);

// do something with classes

free(classes);

The question is then: how do we determine which classes are actually subclasses of some parent class?

The intuitive approach would be to use the isSubclassOfClass: method:

NSMutableArray *result = [NSMutableArray array];
for (NSInteger i = 0; i < numClasses; i++)
{
    if ([classes[i] isSubclassOfClass:parentClass])
    {
        [result addObject:classes[i]];
    }
}

Unfortunately, we can't do this because when considering all classes in the runtime, the method isSubclassOfClass: may not be present on all of them (since isSubclassOfClass: is an NSObject method).

The problem here is that many of the methods that are commonly considered to be implemented by all classes, are actually implemented by NSObject or in an implementation of the NSObject protocol. Some classes, like _NSZombie_ or NSProxy don't derive from NSObject and even if they do implement the NSObject protocol, may do so in unpredictable ways (_NSZombie_ throws an exception for any method and NSProxy forwards many of these methods to its target instead of responding for itself).

So instead of invoking any methods on arbitrary classes we must use the runtime functions. We use class_getSuperclass() to find out the class' superclass and compare this to the parentClass. This gives the complete solution:

NSArray *ClassGetSubclasses(Class parentClass)
{
    int numClasses = objc_getClassList(NULL, 0);
    Class *classes = NULL;

    classes = malloc(sizeof(Class) * numClasses);
    numClasses = objc_getClassList(classes, numClasses);
    
    NSMutableArray *result = [NSMutableArray array];
    for (NSInteger i = 0; i < numClasses; i++)
    {
        Class superClass = classes[i];
        do
        {
            superClass = class_getSuperclass(superClass);
        } while(superClass && superClass != parentClass);
        
        if (superClass == nil)
        {
            continue;
        }
        
        [result addObject:classes[i]];
    }

    free(classes);
    
    return result;
}

The fast and dirty hack approach

The function I just gave is the correct approach to use in a program. But I wouldn't feel like I'd made an interesting post if I didn't also show a completely different, totally unsafe approach too.

This approach will require the Objective-C Runtime 2.0 (that's the one used in Mac OS X 64-bit and on the iPhone). In the new version of the runtime, a class actually contains a direct link to its subclasses.

From objc-runtime-new.h in Apple's opensource repository, a class is declared as follows:

typedef struct class_t {
    struct class_t *isa;
    struct class_t *superclass;
    Cache cache;
    IMP *vtable;
    class_rw_t *data;
} class_t;

with the class_rw_t struct definition:

typedef struct class_rw_t {
    uint32_t flags;
    uint32_t version;

    const class_ro_t *ro;
    
    struct method_list_t **methods;
    struct chained_property_list *properties;
    struct protocol_list_t ** protocols;

    struct class_t *firstSubclass;
    struct class_t *nextSiblingClass;
} class_rw_t;

This means that we can traverse along the firstSubclass and nextSiblingClass members to reach all of the subclasses of a parent class without needing to read every class in the runtime.

A depth-first recursive traversal then gives the full list of subclasses for a class:

typedef void *Cache;
#import "objc-runtime-new.h"

void AddSubclassesToArray(Class parentClass, NSMutableArray *subclasses)
{
    struct class_t *internalRep = (struct class_t *)parentClass;
    
    // Traverse depth first
    Class subclass = (Class)internalRep->data->firstSubclass;
    while (subclass)
    {
        [subclasses addObject:subclass];
        AddSubclassesToArray(subclass, subclasses);
    
        // Then traverse breadth-wise
        struct class_t *subclassInternalRep = (struct class_t *)subclass;
        subclass = (Class)subclassInternalRep->data->nextSiblingClass;
    }
}

However, while highly efficient compared to traversing the whole list of classes, this approach cannot be safely used. According to objc-runtime-new.m, all access to the data member of a class_t must be protected by the runtimeLock — which is inaccessible outside the runtime library itself.

Since the lock is inaccessible, it means that this approach cannot be made safe, since any thread (including threads automatically started by Cocoa) could cause you to crash.

It was a fun experiment and I hope that Apple include a function to do this safely in the future.

Conclusion

The runtime provides no simple method to access the subclasses of a class — even in the newer version of the runtime where this data is actually stored in the class' data structure.

Filtering out classes from the list of all classes presents some interesting problems due to the classes in the runtime which don't derive from NSObject. Ultimately though, once you're aware of the limitations of using the full class list, getting subclasses is fairly simple.

Even though it isn't data you normally need, it would be nice if Apple had decided to offer access to the subclass data in the new version of the runtime. It is more efficient and allows the list to be generated in sorted order if required.

Read more...

Quality control in application development without unit testing

In my last two posts, I've shown a Mac app with full unit tests and an iPhone app with full unit tests. The reality though is that I do not write or test code this way. In this post, I look at why so few applications are actually developed using unit tests. I'll also look at the alternate approaches — both manual and automated — that are normally used to maintain high quality and low bug rates in application development.

Introduction

I have many friends who love unit tests. They never write code without writing the unit tests first. And they love it; the thought of writing code without unit tests seems foolish and scary to them. Test-first approaches ensure that they never write anything that isn't specified and unit tested.

This post is not to dissuade these people from unit tests. If you already know that you know and love unit tests, then stick with them.

But even unit testing advocates should remember that unit tests are about code-level specification and working within certain types of development methodology — they do not attempt to catch all bugs.

This post is primarily for people who have heard of unit tests and are wondering if they are an efficient way to maintain product quality and lower bug rates.

The short answer is: no. If you simply want to lower bug counts, the best and most efficient approach is through system tests.

Difficulties of unit testing for application development

Depending on your development methodology, unit tests can serve a number of different functions. But if used purely to detect bugs then they are a very high cost approach for low return.

Unit tests don't test the overall program, only isolated units. This creates many holes (integration, timing, re-entrancy, inter-module communication) that need to be tested another way. These limitations are not unique to application development so I'll focus instead on something that is: the difficulty of isolating your units for testing.

Advocates of unit testing claim that you can follow good program design and reduce coupling and integration, making tests easy to write.

Unfortunately, in application development, that's not exactly possible.

As I noted when I was discussing iPhone application design, the code that you write in a Cocoa application is mostly controller classes — since the model and view classes are so reusable, you rarely need to write them yourself.

Controller classes are the classes which link your program together — they join model and view elements together to make your program. The entire purpose of these classes is coupling. The unit tests I wrote for the WhereIsMyMac program were approximately three times the size of the original code due to large numbers of mock objects and tricky hackery to intercept calls into the Cocoa framework — and this was a simple sample application, it could easily be much worse. Every mock object reduces the accuracy of the test by reducing the reality of the test environment and the risk of false positives or negatives in test results.

Since controller classes are so common in applications and the main role of a controller class is to join different objects together, unit testing suffers from the following serious problems in application development:

  • Requires huge amounts of mocking code and other fakery (time consuming).
  • The resulting tests are far removed from the integrated reality, making false positives and false negatives highly likely and leaving large holes that are simply not tested.

System testing methodologies for application development

The best approach for testing an application is to accept the coupled, integrated, timing dependent, environment dependent nature and test everything in place. The idea is simple: test the complete program in the exact manner (or as close as possible) in which you expect it to be used.

Tests which operate in this manner are classed as "system tests".

The reason why system tests are the most efficient and accurate form of product testing is simple: if you want to ensure low bug rates for the user, test the program in the same way that the user will use it; don't test a fake environment and don't test at levels the user can't access.

Types of system testing

There are lots of ways to system test your code. In order from most important to least important for application development:

  1. Sanity testing
  2. User interface testing
  3. API testing
  4. Regression testing
  5. Performance testing
  6. Load testing
  7. Scalability testing

I'm not going to give full definitions of these tests here — I'm simply going to discuss their importance to applications and related projects. Follow the links to read more about each one.

Every good application has point 2 and most have 3 and 4 as part of their regular testing methodology — sanity checks occur as part of the development methodology (run your code before you commit) and fall outside formal testing.

The important step in making system tests work is to keep them formalized. If the test is not automated, then it should be formally documented so that each of the steps is correctly run during a testing phase. A basic or ad hoc approach to any of these points doesn't count though: proper quality comes from rigorous and clearly defined testing approaches.

User interface testing

User interface testing is normally performed using testing matrices. In its simplest form, this means a document containing all user interface test cases in a table — a spreadsheet, workprocessor or TextEdit document, it doesn't really matter as long as it is documented.

The rows in the table are all steps in operating your program (selecting menu items, operating buttons, perform edit operations). Every single user interface element in the program should be tested and most should be tested multiple ways to account for different expected effects.

The columns are all environmental differences between runs. Different operating system versions, different computers, different installation settings or different builds of the application (lite, demo or full).

Each cell in the table should then contain the observed result and an indication of whether this is a success or failure according to the specification.

If you can automate the process, then great — there are software tools that will help with this on the Mac. If you can't — yes, user interface testing will get manual and tedious but ultimately, it is the only way to guarantee that the program works as expected. Most of the programs you use have simply had someone sitting in front of them, progressively clicking all the buttons and using all the controls — for every test case in every version.

You need to have the whole table documented and it needs to cover every element of the user interface. The purpose is to remind you to test everything (bad luck will ensure that failure to test a row will guarantee that it contains uncaught bugs).

API testing

In theory, user-interface testing should test everything in a user facing application. It may seem inefficient to suggest API testing (which is normally done for libraries and code modules).

There are three reasons that any substantial application would want API tests:

  • Human testers of user-interfaces are lazy, forget or make mistakes
  • API tests are automated and can be run at build-time or as part of continuous integration
  • You can test issues that may not be obvious during user testing (like data coherency)

For user applications though, it presents the problem that you must create an API layer that can actually be programmatically tested. The common approach is to separate the "model" of your application into a separate module with an API layer.

API testing is similar to unit testing in many respects:

  • it is automated
  • can be implemented using OCUnit or similar libraries
  • can be used to develop your application using test-first methodologies

The difference is that API tests do not separate the units within the module. API tests test the whole module in-place, fully integrated. This means that the API tests can be quite removed from the implementation details — which has the advantage that they can be more easily written by a separate programmer or test engineer, freeing up programming resources. API tests also aim to be optimally efficient: they only test inputs and outputs and don't care about how intermediary steps in the transformation occurs.

API tests attempt to provide a realistic environment and data but do suffer from some of the same limitations as unit tests in that some parts of the environment must be synthetic (the full application will not be present during testing).

Regression tests

After you're happy that your program is working as intended, regression tests ensure that subsequent work doesn't screw it up.

These are tests are normally used for programs that produce a file output. Their operation is basically: run the regression test and compare the output to the "known good" result which was saved previously. If the output changes unexpectedly, the test fails.

For many file producing projects (like a few major open-source projects specializing in video codecs, PDF renderers, DVD authoring packages) this is the only test they include. However, it's all that these types of project require: a good set of regression tests should have high code coverage (exercise almost all of the program) and perform most of the work of API testing too.

Regression tests can be human or automatically driven. Applescript can drive regression tests in full applications or OCUnit can be used to drive regression tests through an API layer.

Specification

As I hinted previously, there are a few roles served by unit tests that system tests handle. One is that unit tests are a form of specification for the code. You can use the tests as a way of learning what the code is supposed to do and in some cases for sketching out how future code should operate.

API testing can certainly replicate the unit testing specification at the API level; again, API tests are like an interface optimized application of unit tests.

However, I prefer the old-school approach: document your code with comments. Not with a line or two inside the method but full comment blocks on every single method documenting conditions on all parameters and the return parameters and effects of the method. Like test-first development, it is common to write comments first (specify the functionality of the method, then write it, asserting pre and post conditions if desired).

I use a customized version of Xcode's "Script Menu→HeaderDoc→Insert @method comment" to automate the creation of the comment block's formatting from the method prototype.

If the comments at the start of each method are not clear enough for documentation (think about Apple's Cocoa API documentation) then you're not doing your commenting job.

Conclusion

Unit testing an application is filled with difficulties and problems. In my development style, I consider the time cost of unit testing an application outweighs its benefits — especially since a unit tested application still requires system tests like user-interface and regression tests for proper validation.

Regardless of whether you use unit tests, formalized system testing — either automated or manual and methodical — is required to fully validate an application and ensure the lowest possible low bug rates.

The most efficient approach is to test the interface that the program exposes in the exact way that the user will use it. For user interface apps, this means user testing matrices. For applications with a lot of model code, this means API tests. For document producing applications, this means regression tests. In many cases though, a combination of all three is best.

Most of these approaches require that you be disciplined. You need to comment your code. You need to maintain user interface testing matrices. You need to refactor your model layer so it has an interface that can be tested automatically. You need to measure the code coverage of your regression tests.

I know these things are tedious. I know that no one wants to write test plans, test documents and API tests. But if you love your program and you want it to suck less, this is work that needs to be done.

Read more...