Maintaining an Open Source Project: Project Organization

By Jacob Wenger

I have accumulated a fair amount of experience working on open source projects during my time at Firebase, a backend-as-a-service company which makes it easy to build realtime applications. We maintain several open source libraries, most of which are written in JavaScript. Over time we devised a common project structure and tool set across them. In this series of tutorials, I will share my learnings and introduce you to the tools that make my life easier on a daily basis. In order to actually show you how everything works, we will walk through the development of OSCalc, a very basic open source calculator written in JavaScript.

Motivation

There is a lot of cognitive overhead required to maintain even a single open source project. Most turn into a mess of files and tools which makes it hard for contributors and users alike. At Firebase, we realized we needed to standardize the structure of our open source projects in order to reduce the amount of time we spent figuring out how things work. Simplifying things to a set of common features also had the dual benefit of making is easier for others to understand and contribute to our projects.

Tools like gulp, Karma, and Travis (all of which I will discuss in this tutorial series) can save you minutes to hours per day, which adds up quickly. However, with the plurality of software out there, it is often hard to find the signal through the noise. Even though at Firebase we are continuously integrating new tools into our project pipeline, I think we have found a good set of tools which reduces the burden of developing software. I will walk you through how we organize, develop, test, and deploy our open source projects so you can just focus on making your library. Let's jump right in.

Creating Your Project

The first step in starting an open source project (after coming up with an idea of course) is actually creating a place to store the project. You should use a version control system such as Git to track changes to your project over time and easily share code with others. I would suggest using GitHub to host your project's code repository. This has become the de facto standard in the industry and the user experience is great. While sometimes you need to interact with git through the command line, I have found that GitHub's desktop program (for Mac or Windows) reduces the chance of the repository getting messed up. GitHub also provides several features for managing your open source project, including a code review tool and an issue tracker.

If you would rather not use GitHub, I have also had a good experience with BitBucket, which supports both Git and Mercurial.

Directory Structure

Once you have created your repository, it is time to start scaffolding your project. We will start with a top-level directory structure which looks like this:

We will walk through each item individually. I created a GitHub repository for OSCalc which you can use to refer to any of the directories or files shown above. I will update this repository as this tutorial series goes on.

src/

The src/ - short for "source" - directory contains the JavaScript files which constitute the meat of our library. This is where most of your coding will take place. These source files will be used to generate the final distribution files.

dist/ (not checked in)

The distribution directory contains your library's output files which will actually be consumed by your users. The dist/ folder typically contains two files: a minified) and an un-minified version of the library. The un-minified version is useful during debugging while the minified file is intended for production use. Minifying your library reduces the bandwidth needed to download it, improving page load times and memory usage, especially on mobile devices. As we will see in the next part of this tutorial series, the dist/ folder is generated from the src/ folder and therefore is not required to be checked into your repository.

tests/

This directory contains your library's test suite. Testing is an integral - yet often neglected - part of writing open source software. Having a comprehensive, reliable test suite provides you with enough credibility for others to use your software in their production applications. A whole part of this tutorial series will be devoted to testing, so that is all I will say about it for now.

examples/

No open source project is complete without examples of how to use it. I will not touch on this too much in this series as examples differ from project to project. That being said, make sure to include very basic examples as well as at least one full-fledged example application showing what is possible with your library.

LICENSE

Choosing a license is a tiny but important part of creating open source software. All of our open source projects at Firebase are released with the permissive MIT license which lets others do whatever they want with the software as long as they provide an attribution back to us. There are several other levels of open source software licenses and you should familiarize yourself with at least the most common ones. GitHub created a handy website to choose which license is best for your project.

README.md

This file is the face of your project and is the first thing that people see when they visit your project's GitHub repository. The .md extension means this is a Markdown file. This file is parsed into formatted HTML every time a user visits your project's repository on GitHub.

Since this is often the first and only thing a developer sees when using your library, it is important that it contains enough information for them to get started. I would suggest providing at least the following information in your project's README.md file:

  • An explanation of what your project does and your motivations behind making it
  • Instructions on how to download and install your library
  • A full API reference
  • Links to live examples
  • Instructions on how others can contribute to your project

package.json

npm is a JavaScript package manager which simplifies the process of working with application dependencies. It is often installed automatically alongside Node.js. npm uses a package.json file to specify your project's dependencies. Developers can later use this file to install your dependencies into a node_modules/ directory by simply running npm install. This is much simpler than the alternative of checking all your dependencies into your project's repository. It also makes it easy to upgrade the versions of certain software that your project is reliant upon because all your dependencies are listed explicitly in one file.

bower.json

Bower is similar to npm in that it is a JavaScript package manager, but it focuses strictly on front-end libraries, such as jQuery or Angular. You can install Bower via npm by running npm install -g bower. Similar to npm, your Bower dependencies will be downloaded into the bower_components/ directory after running bower install.

We will use both package managers to keep track of project dependencies. We will also publish the final library to both npm and Bower so that other people can easily consume it. I will have more to say about both npm and Bower during future parts of this tutorial series.

.gitignore

This file tells git which files and directories not to check into source control. By default, all files are included in your repository. However, generated files (distribution files, code coverage analyses, node_modules/, bower_components/, etc.) often unnecessarily clutter up repositories. Other files may contain sensitive information which you do not want to share publicly. You can prevent these files from being included in your repository by simply listing them in a .gitignore file. The included files will still exist locally, but they will not be shared with others.

Conclusion

As with writing any type of software, it is important to start with a good foundation when developing an open source project. This includes using the right tools to make your job easier and organizing the project in such a way that you - and whoever else wants to contribute - can jump in and understand how things are structured.

Now that we have laid our scaffolding, we can get into the fun part of actually writing our library and making use of the many tools available to us. In the next part of this tutorial, we will cover how to automate common development tasks with a task runner called gulp. See you then!