Software Development Cycle

Although in classes, one often works on individual projects, or in small autonomous teams on group projects, this is atypical of software development projects. Product development is an organizational activity (for better or worse). Usually a project will have several developers and a project manager who must work together to produce the product. In addition, there are external influences from marketing, other business and product groups, etc. Even in open source projects, several developers must cooperate and coordinate to develop larger systems, and there is usually someone coordinating the efforts to avoid redundent work and ensure consistency.

Although it has been going on for decades. Software development and project management is still an art and a mystery. Some 60% of projects are cancelled before completion (though it is impossible to tell how many should never have been started in the first place). This, in part, accounts for the popularity of programming methodologies such as OOP, design methodolgies such as UML, design patterns, etc., and project management tools and systems. Managers and businesses are always looking for the magic bullet to make projects work all the time.

In my humble opinion, the real things you need are: A clear idea of what you are trying to build and why, and a small team of very good developers. Unfortunatly, both of these are scarce.

Stages in project development (and their outputs).

[Note: Design, which might be considered a separate phase, is here split between specification and implementation. The design processes these phases is sometimes quite different, or at least performs by different groups.]

Although there is dependency in these phases. They are often running simultaneously. Especially once project is implemented, feature requests, design changes, and bug fixes are happening all continuously.

Each stage generally includes review processes.

Specification

The specification phase usually consist of deciding and documenting exactly what the product is supposed to do, and usually includes the high-level architecture of the system in terms of sub-system (modules, components, etc) and the interfaces between them. This phase typically solicits input from potential customers, users, marketing, sales, as well as other parts of the engineering organization.

The design and architecture aspects of this phase are beyond the scope of this course. The issues at this stage are the design and specification of the system architecture and the interfaces between both the sub-system of the product and between the product and the outside world.

What makes this design phase particularly difficult is that

Again, the problems of this phase are beyond the scope of this course. For this reason, in Problem Set 3, we are implementing an architecture in which most of the design work has been done. There is a published specification and several reference implementations to compare against.

Implementation

The rest of the course has been above OOP design and programming and how to implement a given specification. Here we want to discuss the social and organizational aspects of program development. We will only comment that there is that boundary between the design and implementation phases depends on the design moethodology, the properties of the team and manager, and the culture and policies of the larger organization. In some projects, the design phase is very detailed to the point of specifying every class and it's methods. Implementation is then a matter of coding. In other projects, the design is specified at a much higher level, and the detailed implementation left to developers. In either case, the high level modules and their interactions and interfaces should be worked out in the design phase. These can then be distributed among the development team for implementation.

Once a project reaches a certain size or includes more than one developer, a number of new issues arise.

Source Control

These days any non-trivial project uses some sort of source control system to coordinate multiple developers. There are a variety of popular systems, some proprietary, some public. Some a real hassle to use, some relatively painless.

CVS is a source control system that is relatively popular in both open-source and commercial projects, probably because it is open-source, free, reliable, available on many platforms, and relatively painless to use. CVS differs from some other systems in that developers do not explicitly lock the files that they are working on (which prevents other developers from changing them). rather each developer can check out a complete copy of the source tree and modify any or all of it. (This an important feature; it is very frustrating to find that the file you need to modify to continue has been locked by someone who has left for a long weekend.).

CVS maintains a central repository of source files and keeps a full version history so that any previous state of the project can be extracted. Developers interact with the repository through a series of commands (there are both command-line and GUI interfaces to CVS, though they implement the same commands). Amount these commands are:

More on CVS can be found at www.cvs.com (???).

Build process

Compiling, linking and exprting a project may sound like a trivial task, but with a multi-person project involving tens to hundreds of files, it becomes less so. Particularly if multiple versions or multiple target platforms are required. Often building a project requires specifing many command options to compilers, linkers, resource compilers, etc. The problem becomes more acute when the development group is not responsible for ongoing builds and maintainance, which is often the case. If the project is open course, the build system must be also open (or at least widely available) and easy to use. Again there are several systems to help automate this process. Most IDE's will provide some sort of build functionality (and many provide source control). Microsoft's Visual Studio is a capable effort in this area. The problem with many of these is that they are proprietary and non-portable.

The javac compiler purports to be a build system, but at present I believe it is insuffucient for many projects' needs. There is a project at Apache called ANT which claims to be a build system for Java.

One popular open build system is Make, developed originally for UNIX but now widely available, albeit in somewhat incompatable versions. Major properties of make:

Sample make file.


# Use variable to hold compile flags
JFLAGS = -classpath mypath

# Variable to hold the list of class files we need to produce
CLASSES = Complex.class \
          Polynomial.class \

# main target, causes all files in CLASSES to be made
all: $(CLASSES)

# Rules to produce each Complex.class
# Rules consist of target: dependencies
#                          commands to execute
#
Complex.class: Complex.java
       javac $(JFLAGS) Complex.java    

# General rules to convert .class to .java
%.class:%.java
      javac  $(JFLAGS) @<

# A target to remove non-source files
clean:
   rm *.class

Another build-related issue is how to organize source and object files. For a small project, everythings can live is one directory. However, for a larger project, this becomes cumbersome. There is no general rule for organizing files, though creating a separate directory for each module is usually a good idea.

Java packages
Java intermingles class/method namespace issues with directory organization (at least for class files). A classes in a package named 'foo' will be be expected in subdirectory 'foo' relative to the classpath. (This is probably due to an ill-concevied attempt to impose a Java-centric class oriented view on users and developers, it ends up being simply annoying until eliminated with .jar files.) At any rate, this imposes some constraint on how object files are organized. It is probably a good idea to keep source files in the same hierarchy.

Periodic builds
In a large on-going project, it is generally a good idea to build the latest version of system (as it exists in the source repository) periodically (usually nightly). This is checks that no-one has check in code that is incompatible with the rest of the system (eg changed an interface). If the nightly build fails, a search ensues for the guilty party.

Documentation

Documentation is an integral part of the software development process. One can identify several classes of documentation depending on the intended audience and level of detail.

In general, user and installation documentation is written by professional tech-writers (if you are lucky) with input from the designers and developers, as well as marketing and sales. We will not discuss this further.

Module level documentation is critical both for successful project completion (as it specifying design and functionality) and future maintainance. It can also form the main body of User-level API documentation. It should include, at minimum, a description of each of the classes and interfaces, what they are supposed to represent and do, and descriptions of each of their methods. It should be suffient for a developer unfamiliar with the project to user the module without examining the source code.

Implementation documentation is similar to module documentation but at a lower level. The degree of implementation documentation required is more flexible. In general, classes and important methods should be documented. An overview of the implementation should also be provided covering data representation and any algorithms used. As for in-line documentation, the code plus comments should be a readable description of what the code is doing; it should tell it's story (as well as compiling to a working program). If the code is straightforward and clear, minimal comments are required. If the implementation is in any way tricky or obscure (eg for optimization), provide an explanation of what is going on.

Testing

Debugging

I would like to conclude this lecture with some hints on debugging.