Recently saw Open Source Courseware -- Evaluation and Rating referenced again, this time on elearnspace, and it made me think about what I look for in an open source content management system. A while back, I looked at the open source courseware and determined that, excluding the classroom admin features (as a writing teacher, I don't need the grading & quizzing modules), PostNuke could do the same things and even more. When I used it in the classroom, it did end up proving a successful alternative to Blackboard, both with my f2f class last fall and the online class this past spring. Now I feel like I can never go back to proprietary courseware.

Meanwhile, in thinking about the article, I thought it would be useful to lay out some of my ideas about evaluating content management systems in preparation for our work at Open Education.

Development/Community

Open source software is not like software out of a box. Reading user reviews, reviewing site feature lists, and looking at features and trying a quick demo on sites such as cmsinfo and opensourceCMS are typically not enough. Since open source projects are typically volunteer based, I think it important to understand the developers and the community that they foster.

Examine the mailing lists and forums. Look for what the developers are implementing in the next version. Consider the project base website and evaluate the interaction that occurs within the community.

• By reading the mailing lists, looking at the forums, and seeing how developers react to users, it's possible to see what kinds of new ideas they embrace, and what kinds of old ideas they develop further. Try categorizing the focus of most of their interests. From there, project where the project seems to be headed long term.
• Is the development community open? Do they have a publicly accessible mailing list? Do they have a community site which demonstrates their product in which they field user concerns? What is the tone of the way that they respond to user concerns? Is CVS world readable? Do they have a standard API with documentation? Better to find a development community that seems to invite other contributors, rather than one that stays behind locked doors. This is particularly true if the software needs additional modules or changes to the core to suit your needs.
• Who runs the project? From my limited experience of examining CMS projects, and from thinking about Linux and other examples, the better projects seem to have a lead developer as the leader of the entire project.
• Who is/are the core developers and do they seem to work together well? I would personally be cautious about relying heavily on a product in which one main person makes most of the contributions to the core and in which decisions do not ever seem to be made collaboratively. What happens if that person decides not to work on the project any longer? And, if there's lots of flaming disagreement on develpment direction, be prepared for the possibility of a fork. While a fork might result in two useful software platforms, development is likely to be slowed down on both while new communities are shaped.
• How have the developers setup the main site and what do they use the product for on their own sites? Understanding the motivations of the developers--what they want to use the software for--is a good indication of where the software will head in future development
• Note the commercial ties to the project. It may be that development is centered around commercial interests which may not be inline with your needs.

Examining the Software Itself

Closely examining the software includes not only looking at a demo site, but getting hands dirty by investigating add-on modules, configuration settings, and even the code-base itself.

And installing a test site is a must before using the software in an important production environment. In the long term, having a test site is useful for trying out new modules and upgrades before implementing them on the production site.

• Look carefully at the available modules and features. It's important not only to consider whether the platform contains modules x, y, and z that you need, but also look at modules a,b, and c that you do not need. Are these the popular modules on the development site? Are most other sites in the user community more interested in a,b, and c, too? If so, the features that you like may not get much attention in later builds. If they are not popular modules, then it's also possible that the module maintainer may lag behind updating the module for new releases.
• Make sure to test any add-on modules (those modules that are not part of the core installation provided by the project) which are important to your use before committing to a platform. Since add-on modules are typically not maintained by the core development team, they could be buggier, under maintained (which means they may not work with the current version) or difficult to install and setup.
• Consider the consistency of the GUI experience. For example, will the user that has just learned how the stories/weblog submission process works find that the forums GUI is an entirely different interface? This could make users less likely to use the second module, as well as creating a less-unified feel to the site.
• Which ties into the previous point. Examine add-on modules to see if they are merely ports of another project. Sometimes these modules are radically different in interace design. Others, such as add-on forums, may not integrate the user database into the forum module and require that users register and login to the module separately.
• Is the software a fork of another project? Or has it been designed from the bottom up? A software which is a fork may have more opportunities for new add-on modules as users will often obtain modules from the other project and modify them to work. On the other hand, a fork of a project represents a break in a community, which itself does not bode well for long term productivity. It could also be that the current use was not the intended use and the code base could be in need of heavy revision; a project built from the ground up with a clear vision may be a better designed piece of software.
• In evaluation, don't over-prioritize content management features over the community-building aspects of the software. A site which better enables community interaction is likely to encourage production more than one which has better content management features.
• And, in my opinion, the project should be copyleft, not merely open source, to be inviting to the most number of contributors.

Last revised 22 March 2004.