Chapter 4. The Node Module System
Node’s basic implementation is kept as streamlined as possible. Rather than incorporate every possible component of use directly into Node, developers offer additional functionality via Node’s modules.
Node’s module system is patterned after the CommonJS module system, a way of creating modules so that they’re interoperable. The core of the system is a contract that developers adhere to in order to ensure that their modules play well with others.
Among the CommonJS module system requirements implemented with Node are:
Support is included for a
require
function that takes the module identifier and returns the exported API.The module name is a string of characters, and may include forward slashes (for identification of path).
The module must specifically export that which is to be exposed outside the module.
Variables are private to the module.
In the next several sections, we’ll see how Node adheres to these requirements.
Loading a Module with require and Default Paths
Node supports a simple module loading system: there is a one-to-one correspondence between the file and module.
To include a module within a Node application, use the require
statement, passing in a string with the
identifier for the module:
var http = require ('http');
You can also just include a specific object, rather than all objects, from a module:
var spawn = require('child_process').spawn;
You can load core modules—i.e., those native to Node—or
modules from the node_modules folder just by providing the
module identifier, such as http
for the
HTTP module. Modules not part of core, or not included in the node_modules folder, should include forward
slashes to indicate the path. As an example, Node expects to find the
module named mymodule.js in the same
directory as the Node application in the following require
statement:
require ('./mymodule');
Or you can use the full path:
require ('/home/myname/myapp/mymodule.js');
Module files can have either a .js, .node, or .json file extension. The .node extension assumes that the file is a compiled binary, not a text file containing JavaScript.
Node core modules have higher priority than external modules. If
you’re trying to load a custom module named http
, Node loads the core version of the HTTP
module. You’ll have to provide either a different module identifier, or
you’ll need to provide the full path.
Earlier I mentioned the node_modules folder. If you specify the node identifier without providing a path, and the module isn’t a core module, Node first looks for a node_modules folder local to the application, and searches for the module in this folder. If it doesn’t find the module, Node then looks in the parent subdirectory for a node_modules folder and the node, and so on.
If the module is named mymodule
,
and the application is located in a subdirectory with the following
path:
/home/myname/myprojects/myapp
then Node looks for the module using the following searches, in turn:
/home/myname/myprojects/myapp/node_modules/mymodule.js
/home/myname/myprojects/node_modules/mymodule.js
/home/myname/node_modules/mymodule.js
/node_modules/mymodule.js
Node can optimize the search depending on where the file issuing the
require
statement resides. For
instance, if the file making the require
statement is itself a module in a
subdirectory of the node_modules
folder, Node begins the search for the required module in the topmost
node_modules folder.
There are two additional variations of require
: require.resolve
and
require.cache
. The
require.resolve
method
performs the lookup for the given module but, rather than load the module,
just returns the resolved filename. The resolve.cache
object
contains a cached version of all loaded modules. When you try to load the
module again in the same context, it’s loaded from the cache. If you want
to force a new load, delete the item from the cache.
If the item’s path is:
var circle = require('./circle.js');
delete it with:
delete require.cache('./circle.js');
This code forces a reload of the module the next time a require
is called on it.
External Modules and the Node Package Manager
As mentioned earlier, much of the rich functionality associated with Node comes in via third-party modules. There are router modules, modules for working with relational or document database systems, template modules, testing modules, and even modules for payment gateways.
Though there is no formal Node module developer system, developers are encouraged to upload their modules to GitHub. Following are good resources for finding Node modules:
npm registry (http://search.npmjs.org/)
Node module wiki (https://github.com/joyent/node/wiki/modules)
The node-toolbox (http://toolbox.no.de/)
Nipster! (http://eirikb.github.com/nipster/)
The modules are roughly categorized into different types such as the aforementioned routers, database, templating, payment gateway, and so on.
To use a module, you can download the source from GitHub (or wherever the source is located), and then install it manually into your application environment. Most modules provide basic installation instructions, or, at a minimum, you can deduce the installation requirements by examining the files and directories included in the module. However, there is a far easier way to install a Node module: using the Node Package Manager (npm).
Note
The npm site is at http://npmjs.org/. You can find basic instructions on npm at http://npmjs.org/doc/README.html. Essential reading for Node module developers is the Developers section of the npm manual, found at http://npmjs.org/doc/developers.html. For a useful post explaining the differences between local and global installation, see http://blog.nodejs.org/2011/03/23/npm-1-0-global-vs-local-installation/.
Modern installations include npm, but you can double-check for its
existence by typing npm
at the
command line in the same environment that you use to access Node.
To get an overview of npm commands, use the following:
$ npm help npm
Modules can be installed globally or locally. The local installation is the best approach if you’re working on a project and not everyone sharing the system needs access to this module. A local installation, which is the default, installs the module in the current location in the node_modules directory.
$ npm install modulename
As an example, to install Connect, a very popular middleware framework, use the following:
$ npm install connect
npm not only installs Connect, it also discovers its module dependencies and installs them, too, as shown in Figure 4-1.
Once it’s installed, you can find the module in your local directory’s node_modules directory. Any dependencies are installed in that module’s node_modules directory.
If you want to install the package globally, use the -g
or --global
option:
$ npm -g install connect
These examples install packages that are registered at the npm site. You can also install a module that’s in a folder on the filesystem, or a tarball that’s either local or fetched via a URL:
npm install http://somecompany
.com/somemodule
.tgz
If the package has versions, you can install a specific version:
npm install modulename
@0.1
Note
npm can also work with Git, as demonstrated in the Appendix A.
You can even install an old friend, jQuery:
npm install jquery
Now you can make use of a familiar syntax in your Node application development.
If you’re no longer using a module, you can uninstall it:
npm uninstall modulename
The following command tells npm to check for new modules, and perform an update if any are found:
npm update
Or you can update a single module:
npm update modulename
If you just want to check to see if any packages are outdated, use the following:
npm outdated
Again, you can run this command against a single module.
List installed packages and dependencies with list
, ls
, la
, or ll
:
npm ls
The la
and ll
options provide
extended descriptions. The following is the text I get running
npm
ll
in my Windows 7
machine:
C:\Users\Shelley>npm ls ll npm WARN jsdom >= 0.2.0 Unmet dependency in C:\Users\Shelley\node_modules\html5 C:\Users\Shelley ├── async@0.1.15 ├── colors@0.6.0-1 ├── commander@0.5.2 ├─┬ connect@1.8.5 │ ├── formidable@1.0.8 │ ├── mime@1.2.4 │ └── qs@0.4.1 ├─┬ html5@v0.3.5 │ ├── UNMET DEPENDENCY jsdom >= 0.2.0 │ ├── opts@1.2.2 │ └─┬ tap@0.0.13 │ ├── inherits@1.0.0 │ ├── tap-assert@0.0.10 │ ├── tap-consumer@0.0.1 │ ├── tap-global-harness@0.0.1 │ ├── tap-harness@0.0.3 │ ├── tap-producer@0.0.1 │ ├── tap-results@0.0.2 │ ├─┬ tap-runner@0.0.7 │ │ ├── inherits@1.0.0 │ │ ├── slide@1.1.3 │ │ ├── tap-assert@0.0.10 │ │ ├── tap-consumer@0.0.1 │ │ ├── tap-producer@0.0.1 │ │ ├── tap-results@0.0.2 │ │ └── yamlish@0.0.3 │ ├── tap-test@0.0.2 │ └── yamlish@0.0.2 └─┬ optimist@0.3.1 └── wordwrap@0.0.2
Note the warning about an unmet dependency for the HTML5 module. The HTML5 module requires an older version of the JSDOM library. To correct this, I installed the necessary version of the module:
npm install jsdom@0.2.0
You can also directly install all dependencies with the -d
flag. For instance, in
the directory for the module, type the following:
npm install -d
If you want to install a version of the module that hasn’t yet been uploaded to the npm registry, you can install directly from the Git repository:
npm install https://github.com/visionmedia/express/tarball/master
Use caution, though, as I’ve found that when you install a not-yet-released version of a module, and you do an npm update, the npm registry version can overwrite the version you’re using.
To see which modules are installed globally, use:
npm ls -g
You can learn more about your npm installation using the config
command. The
following lists the npm configuration settings:
npm config list
You can get a more in-depth look at all configuration settings with:
npm config ls -l
You can modify or remove configuration settings either by using a command line:
npm config deletekeyname
npm config setkeyname
value
or by directly editing the configuration file:
$ npm config edit
Warning
I would strongly recommend you leave your npm configuration settings alone, unless you’re very sure of a change’s effect.
You can also search for a module using whatever terms you think might return the best selection:
npm search html5 parser
The first time you do a search, npm builds an index, which can take a few minutes. When it’s finished, though, you’ll get a list of possible modules that match the term or terms you provided. The search terms html5 and parser returned just two modules: HTML5, an HTML parser that includes support for SVG and MathML; and Fabric, an object model with support for HTML5 Canvas and an SVG-to-Canvas parser.
The npm website provides a registry of modules you can browse through, and an up-to-date listing of modules most depended on—that is, modules most used by other modules or by Node applications. In the next section, I’ll cover a sampling of these modules.
Note
I’ll cover other npm commands later in this chapter, in the section Creating Your Own Custom Module.
Finding Modules
Though Node.js has been active only for a few years, it’s already attracted a large body of work. When you look at the Node.js modules wiki page, you’ll find a significant number of modules. The good thing is, you can find a lot of useful modules that implement the functionality you need. The bad news is, it’s difficult to determine which module to use—in other words, which modules are “best of breed.”
Using a search tool like Google can give you a fair idea of which modules are popular. For example, it quickly became apparent when I was exploring middleware and framework modules that Connect and Express were very popular.
In addition, when you look at the GitHub registry for the item, you can see if it’s actively supported and up to date with the current Node installation. As another example, I was checking out a tool named Apricot, which does HTML parsing and is recommended in the Node documentation, but then I noticed it hadn’t been updated for some time, and when I tried to use the module, I found it didn’t work with my installation of Node (at least, not when this book was written).
Note
Many of the modules provide example applications, and a quick test of these will let you know if you can use the module in your current environment.
As mentioned, the Node documentation site does provide a listing of recommended third-party modules, starting with the npm, which is now incorporated into the Node installation. However, the npm website and its module registry provide us with a better view of what modules are used in most applications.
At the npm registry page, you can search for modules, but you can also review a list of the “most depended on” modules, which are modules used either in other modules or in Node applications. At the time of this writing, the top modules were:
I’ll cover several of these modules in future chapters, but I want to cover three now—both because they give us a chance to better understand how Node works, and because they are especially useful. They are:
Colors
Optimist
Underscore
Colors: Simple Is Best
Colors is one of the simpler modules. You can use it to
provide different color and style effects to the console.log
output, and
that’s it. However, it’s also a good demonstration of an effective
module because it is simple to use, focuses on providing one service,
and does a good job with that one service.
Testing how a module works is a great reason for using REPL. To try Colors, install it using npm like so:
$ npm install colors
Open a new REPL session and include the colors
library:
> var colors = require('colors');
Because the Colors module is included in the current location’s node_modules subdirectory, Node is able to find it rather quickly.
Now try something out, such as the following:
console.log('This Node kicks it!'.rainbow.underline);
The result is a colorful, underlined rendering of your message. The style applies only for the one message—you’ll need to apply another style for another message.
If you’ve worked with jQuery, you recognize the chaining used to
combine effects. The example makes use of two: a font effect, underlined
, and a font color, rainbow
.
Try another, this time zebra
and bold
:
console.log('We be Nodin'.zebra.bold);
You can change the style for sections of the console message:
console.log('rainbow'.rainbow, 'zebra'.zebra);
Why would something like Colors be useful? One reason is that it enables us to specify formatting for various events, such as displaying one color for errors in one module, another color or effect for warnings in a second module, and so on. To do this, you can use the Colors presets or create a custom theme:
> colors.setTheme({ ....... mod1_warn: 'cyan', ....... mod1_error: 'red', ....... mod2_note: 'yellow' ....... }); > console.log("This is a helpful message".mod2_note); This is a helpful message > console.log("This is a bad message".mod1_error); This is a bad message
Note
Find more on Colors at https://github.com/Marak/colors.js.
Optimist: Another Short and Simple Module
Optimist is another module focused on solving a specific problem: parsing command options. That’s it, that’s all it does—but it does it very well.
As an example, this simple application uses the Optimist module to print command-line options to the console.
#!/usr/local/bin/node var argv = require('optimist').argv; console.log(argv.o + " " + argv.t);
You can run the application with short options. The following
prints the values of 1
and 2
out to the console:
./app.js -o 1 -t 2
You can also process long options:
#!/usr/local/bin/node var argv = require('optimist').argv; console.log(argv.one + " " + argv.two);
and test with the following, resulting in a printout of My Name
:
./app2.js --one="My" --two="Name"
You can also use the Optimist module to process Boolean and unhyphenated options.
Note
Read more about Optimist at https://github.com/substack/node-optimist.
Underscore
Install the Underscore module with:
npm install underscore
According to the developers, Underscore is a utility-belt library for Node. It provides a lot of extended JavaScript functionality we’re used to with third-party libraries, such as jQuery or Prototype.js.
Underscore is so named because, traditionally, its
functionality is accessed with an underscore (_
), similar to jQuery’s $
. Here’s an example:
var _ = require('underscore'); _.each(['apple','cherry'], function (fruit) { console.log(fruit); });
Of course, the problem with the underscore is that this character
has a specific meaning in REPL. No worries, though—we can just use
another variable, us
:
var us = require('underscore'); us.each(['apple','cherry'], function(fruit) { console.log(fruit); });
Underscore provides expanded functionality for arrays, collections, functions, objects, chaining, and general utility. Fortunately, there’s also excellent documentation for all of its functionality, so I’ll forgo detailing any of it here.
However, I do want to mention one nice capability: a controlled
way to extend Underscore with your own utility functions, via the
mixin
function. We can
quickly try this method, and the others, in a REPL session:
> var us = require('underscore'); undefined > us.mixin({ ... betterWithNode: function(str) { ..... return str + ' is better with Node'; ..... } ... }); > console.log(us.betterWithNode('chocolate')); chocolate is better with Node
Note
You’ll see the term mixin used in several Node modules. It’s based on a pattern where properties of one object are added (“mixed in”) to another.
Of course, it makes more sense to extend Underscore from a module that we can reuse in our applications, which leads us to our next topic—creating our own custom modules.
Creating Your Own Custom Module
Just as you do for your client-side JavaScript, you’ll want to split off reusable JavaScript into its own libraries. The only difference is that you need to take a couple of extra steps to convert your JavaScript library into a module for use with Node.
Let’s say you have a JavaScript library function, concatArray
, that takes a string and an array of
strings, and concatenates the first string to each string in the
array:
function concatArray(str, array) { return array.map(function(element) { return str + ' ' + element; }); }
You want to use this function, as well as others, in your Node applications.
To convert your JavaScript library for use in Node, you’ll need to
export all of your exposed functions using the exports
object, as shown
in the following code:
exports.concatArray = function(str, array) { return array.map(function(element) { return str + ' ' + element; }); };
To use concatArray
in a Node
application, import the library using require
, assigning the
library to a variable name. Once the library is assigned, you can call any
of the exposed functions in your code:
var newArray = require ('./arrayfunctions.js'); console.log(newArray.concatArray('hello', ['test1','test2']));
It’s not very complicated, as long as you remember two things:
Packaging an Entire Directory
You can split your module into separate JavaScript files, all located within a directory. Node can load the directory contents, as long as you organize the contents in one of two ways.
The first way is to provide a JSON file named package.json with information about the directory. The structure can contain other information, but the only entries relevant to Node are:
{ "name" : "mylibrary", "main" : "./mymodule/mylibrary.js" }
The first property, name
, is the name of
the module. The second, main
, indicates the
entry point for the module.
The second way is to include either an index.js or index.node file in the directory to serve as the main module entry point.
Why would you provide a directory rather than just a single
module? The most likely reason is that you’re making use of existing
JavaScript libraries, and just providing a “wrapper” file that wraps the
exposed functions with exports
statements.
Another reason is that your library is so large that you want to break
it down to make it easier to modify.
Regardless of the reason, be aware that all of the exported objects must be in the one main file that Node loads.
Preparing Your Module for Publication
If you want to make your package available to others, you can promote it on your website, but you’ll be missing out on a significant audience. When you’re ready to publish a module, you’re going to want to add it to the list of modules at the Node.js website, and you’ll also want to publish it to the npm registry.
Earlier I mentioned the package.json file. It’s actually based on the CommonJS module system recommendations, which you can find at http://wiki.commonjs.org/wiki/Packages/1.0#Package_Descriptor_File (though check to see if there’s a more up-to-date version).
Among the required fields for the package.json file are:
name
The name of the package
description
The package description
version
The current version conforming to semantic version requirements
keywords
An array of search terms
maintainers
An array of package maintainers (includes name, email, and website)
contributors
An array of package contributors (includes name, email, and website)
bugs
The URL where bugs can be submitted
licenses
An array of licenses
repositories
An array of repositories where the package can be found
dependencies
Prerequisite packages and their version numbers
The other fields are optional. Still, that’s a lot of fields. Thankfully, npm makes it easier to create this file. If you type the following at the command line:
npm init
the tool will run through the required fields, prompting you for each. When it’s done, it generates a package.json file.
In Chapter 3, Example 3-13, I started an
object called inputChecker
that
checks incoming data for commands and then processes the command. The
example demonstrated how to incorporate EventEmitter
. Now we’re going to modify this
simple object to make it usable by other applications and
modules.
First, we’ll create a subdirectory in node_modules and name it inputcheck, and then move the existing
inputChecker
code file to it. We need
to rename the file to index.js.
Next, we need to modify the code to pull out the part that implements
the new object. We’ll save it for a future test file. The last
modification we’ll do is add the exports
object, resulting in the code shown in
Example 4-1.
var util = require('util'); var eventEmitter = require('events').EventEmitter; var fs = require('fs'); exports.inputChecker = inputChecker; function inputChecker(name, file) { this.name = name; this.writeStream = fs.createWriteStream('./' + file + '.txt', {'flags' : 'a', 'encoding' : 'utf8', 'mode' : 0666}); }; util.inherits(inputChecker,eventEmitter); inputChecker.prototype.check = function check(input) { var self = this; var command = input.toString().trim().substr(0,3); if (command == 'wr:') { self.emit('write',input.substr(3,input.length)); } else if (command == 'en:') { self.emit('end'); } else { self.emit('echo',input); } };
We can’t export the object function directly, because util.inherits
expects
an object to exist in the file named inputChecker
. We’re also modifying the
inputChecker
object’s prototype
later in the file. We could have
changed these code references to use exports.inputChecker
, but that’s
kludgy. It’s just as easy to assign the object in a
separate statement.
To create the package.json
file, I ran npm init
and answered
each of the prompts. The resulting file is shown in Example 4-2.
{ "author": "Shelley Powers <shelleyp@burningbird.net> (http://burningbird.net)", "name": "inputcheck", "description": "Looks for commands within the string and implements the commands", "version": "0.0.1", "homepage": "http://inputcheck.burningbird.net", "repository": { "url": " }, "main": "inputcheck.js", "engines": { "node": "~0.6.10" }, "dependencies": {}, "devDependencies": {}, "optionalDependencies": {} }
The npm init
command doesn’t
prompt for dependencies, so we need to add them directly to the file.
However, the inputChecker
module
isn’t dependent on any external modules, so we can leave these fields
blank in this case.
Note
Chapter 16 has a more in-depth look at the package.json file.
At this point, we can test the new module to make sure it actually
works as a module. Example 4-3 is
the portion of the previously existing inputChecker
application that tested the new
object, now pulled out into a separate test application.
var inputChecker = require('inputcheck').inputChecker; // testing new object and event handling var ic = new inputChecker('Shelley','output'); ic.on('write', function(data) { this.writeStream.write(data, 'utf8'); }); ic.addListener('echo', function( data) { console.log(this.name + ' wrote ' + data); }); ic.on('end', function() { process.exit(); }); process.stdin.resume(); process.stdin.setEncoding('utf8'); process.stdin.on('data', function(input) { ic.check(input); });
We can now move the test application into a new examples subdirectory within the module directory to be packaged with the module as an example. Good practice demands that we also provide a test directory with one or more testing applications, as well as a doc directory with documentation. For a module this small, a README file should be sufficient. Lastly, we create a gzipped tarball of the module.
Once we’ve provided all we need to provide, we can publish the module.
Publishing the Module
The folks who brought us npm also provide a really great source for Node developers: the Developer Guide. It outlines everything we need to know about how to publish our modules.
The Guide specifies some additional requirements for the package.json file. In addition to the fields
already created, we also need to add in a directories
field with
a hash of folders, such as the previously mentioned test and doc:
"directories" : { "doc" : ".", "test" : "test", "example" : "examples" }
Before publishing, the Guide recommends we test that the module can cleanly install. To test for this, type the following in the root directory for the module:
npm install . -g
At this point, we’ve tested the inputChecker
module, modified the package.json package to add directories, and
confirmed that the package successfully installs.
Next, we need to add ourselves as npm users if we haven’t done so already. We do this by typing:
npm adduser
and following the prompts to add a username, a password, and an email address.
There’s one last thing to do:
npm publish
We can provide the path to the tarball or the directory. As the
Guide warns us, everything in the directory is exposed unless we use a
.npmignore
list in the
package.json file to ignore
material. It’s better, though, just to remove anything that’s not needed
before publishing the module.
Once published—and once the source is also uploaded to GitHub (if that’s the repository you’re using)—the module is now officially ready for others to use. Promote the module on Twitter, Google+, Facebook, your website, and wherever else you think people would want to know about the module. This type of promotion isn’t bragging—it’s sharing.
Get Learning Node now with the O’Reilly learning platform.
O’Reilly members experience books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.