November 2016
Volume 31 Number 11
[The Working Programmer]
How To Be MEAN: Taking a Gulp
By Ted Neward | November 2016
.jpg "Ted Neward")
Welcome back, MEANers.
If you read my last column in October, which talked about the “reboot” of the codebase (by using Yeoman to scaffold out the basics and glue code), you might have noticed that a new tool was thrown into the mix without much explanation. I refer, of course, to the “Gulp” tool used to start the server and open the client browser to the scaffolded client application (msdn.com/magazine/mt742874).
If you missed my last column, it’s easy enough to catch up. First, make sure Yeoman and the “angular-fullstack” generators are both installed on your Node.js development environment (along with a local running copy of MongoDB):
npm install –g yeoman angular-fullstack-generator
yo angular-fullstack
Then, after answering questions as to what tools Yeoman should scaffold into place (for this column, the choices made are mostly irrelevant), and after Yeoman has helpfully kicked off an “npm install” to pull down all the runtime and development dependencies, the scaffolder will report that the application is ready to test by running “gulp test” or “gulp start:server.”
Clearly, whatever Gulp is, it’s some kind of build tool, akin in spirit to Make, MSBuild or Ant. But how it works is a little different than any of those three tools, and it deserves discussion as a result.
Just a Little Gulp to Start
While it’s not exactly fair to call Gulp a “build tool” for a language that doesn’t ever get built (remember, ECMAScript is generally intended to be an interpreted language), that’s really the best terminology you have for a tool that’s intended to be run after (or during) development to make sure everything is lined up and ready to go. Perhaps a better name for all of them would be “development automation tool,” because that’s more accurate and would include the act of compiling code and assembling it into a deployable artifact. However, because that’s a mouthful and “build tool” just rather rolls off the tongue, let’s go with the idea that Gulp is a build tool for now.
To get started with Gulp, let’s break away from the scaffolded code from earlier and start from scratch to focus on what (and how) Gulp does what it does. Install the global Gulp command-line tools (“npm install --g gulp-cli”) first. Then, in a new directory, create an empty Node.js project by running the following:
npm init
npm install --save-dev gulp
This will make sure that Gulp is referenced in the package.json file as a developer dependency, so that if you pull down the project you won’t have to remember to install Gulp—it’ll just come along for the ride with the next “npm install” in a fresh environment.
Now, in your text editor of choice, create a new file, called gulpfile.js:
const gulp = require('gulp');
gulp.task('default', function() {
console.log("Gulp is running!");
});
Then, from the same directory, issue the stock Gulp command, which (perhaps unsurprisingly) is just “gulp.” Gulp thinks about it for a second, then comes back with:
[18:09:38] Using gulpfile ~/Projects/code/gulpdemo/gulpfile.js
[18:09:38] Starting 'default'...
Gulp is running!
[18:09:38] Finished 'default' after 142 μs
Not bad. Seems a little overkill at the moment, but not bad.
Gulp Down Some Tasks
Gulp, like most build tools, thinks in terms of “tasks,” and more important, how to track the dependencies between those tasks. So, in the simple Gulpfile here, it sees that there’s one task, called “default” (which is the well-understood convention to mean the task that should be executed if none is specified on the command line), and executes the body of the associated function literal when asked to execute that task. Naming the task differently is trivial:
const gulp = require('gulp');
gulp.task('echo', function() {
console.log("Gulp is running!");
});
It should be apparent that Gulp is just code, so anything that can be done in code can also be done in the body of a Gulp task. This offers up many incredible options, like reading from a database to find elements that need to be code-generated, talking to other online services for configuration, or even just printing out the current date and time:
const gulp = require('gulp');
gulp.task('echo', function() {
console.log("Gulp is running on " + (new Date()));
});
This produces the following:
Teds-MacBook-Pro:gulpdemo tedneward$ gulp echo
[18:16:24] Using gulpfile ~/Projects/code/gulpdemo/gulpfile.js
[18:16:24] Starting 'echo'...
Gulp is running on Wed Sep 14 2016 18:16:24 GMT-0700 (PDT)
[18:16:24] Finished 'echo' after 227 μs
Dependent tasks are simply listed as a string array in between the name of the task and its function literal body, so having task “echo” depending on another task simply looks like this:
const gulp = require('gulp');
const child_process = require('child_process');
gulp.task('gen-date', function() {
child_process.exec('sh date > curdate.txt');
});
gulp.task('echo', ['clean', 'gen-date'], function() {
console.log("Gulp is running on " + (new Date()));
});
Here, the “gen-date” task makes use of a standard Node.js package, “child_process,” to kick off an external tool to write the date to a file, just to prove that you can. Which, honestly, is all well and good, but generally build tools are expected to do something of greater import than just write stuff to the console and figure out the current date and time.
Gulping a Little More
Let’s put a bit more meat into this. Create an index.js file with the following ECMAScript code in it, including the not-quite-good code parts:
// index.js
function main(args) {
for (let arg in args) {
if (arg == "hello")
console.log("world!");
console.log("from index.js!");
}
}
console.log("Hello, from index.js!")
main()
Yes, it’s a bit nonsensical and, yes, it clearly has a few issues with it, but that’s the whole point—it would be nice if there were a tool that could spot some of those issues and report on them. (Dare I say, “Compile-time check”?) Fortunately, such a tool exists in JSHint (jshint.com), but it installs by default as a command-line tool and it would be a pain to have to remember to run it all the time.
Fortunately, this is what a build tool is for. Going back to the Gulpfile, let’s get Gulp to run JSHint over each of the source files (of which there is only one right now) by telling it about the source files in question and then asking it to execute JSHint over each:
// Don't need it in this file, but we need it installed
require('jshint');
require('jshint-stylish');
const gulp = require('gulp');
const jshint = require('gulp-jshint');
gulp.task('jshint', function() {
return gulp.src('*.js')
.pipe(jshint())
.pipe(jshint.reporter('jshint-stylish'));
});
When you run this, the tool will point out that there are some suggested changes to all of the “js” files in the current directory, including the Gulpfile itself. Bah. You don’t really care about checking the Gulpfile, so let’s screen it out of the collection of files to run:
gulp.task('jshint', function() {
return gulp.src(['*.js', '!gulpfile.js'])
.pipe(jshint())
.pipe(jshint.reporter('jshint-stylish'));
});
This will remove “gulpfile.js” from the current list of files before passing it on to the next stage in the pipeline. As a matter of fact, you’ll probably want most code to live in a “src” directory (or “server” and “client” directories, for each side), so add those and any of their subdirectories to the list of files to process:
gulp.task('jshint', function() {
return gulp.src(['*.js', '!gulpfile.js', 'server/**/*.js', 'client/**/*.js'])
.pipe(jshint())
.pipe(jshint.reporter('jshint-stylish'));
});
The “double-star” in each path gets the recursive behavior to pick up any “js” files in each of those subdirectories.
On the surface of it, this is great, but it doesn’t really do much for you: You still have to type “gulp jshint” (or just “gulp” if you tied it into the “default” task as a dependency) by hand each time you want to see what needs fixing. Why can’t you just have the thing run any time the code changes, the way IDEs do?
Sure, do that, as shown in Figure 1.
Figure 1 Continuous Automation Using Gulp
// Don't need it in this file, but you need it installed
require('jshint');
const gulp = require('gulp');
const jshint = require('gulp-jshint');
gulp.task('default', ['watch']);
gulp.task('watch', function() {
gulp.watch(['*.js', '!gulpfile.js', 'client/**/*.js', 'server/**/*.js'],
['jshint']);
});
gulp.task('jshint', function() {
return gulp.src(['*.js', '!gulpfile.js', 'client/**/*.js', 'server/**/*.js'])
.pipe(jshint())
.pipe(jshint.reporter('default'));
});
Now, when you run “gulp,” the command line will simply pause and wait. Gulp is now in “watch” mode, wherein it’ll keep an eye on any of the files passed to the “gulp.watch” call, and if any of those files changes (meaning, they’re saved—Gulp can’t peer inside of text editors, unfortunately), it’ll immediately run the “jshint” task on the complete set of files. And keep watching.
Gulping More
One of the keys to understanding how Gulp tasks process files is buried in the call to pipe. Gulp thinks in terms of “streams,” rather than tasks or files. For example, a simple Gulp task that would copy files from the “src” directory to the “dest” directory would look like this:
gulp.task('copy-files', function() {
gulp.src('source/folder/**')
.pipe( gulp.dest('dest/folder/**') );
});
In essence, the files are each picked up by gulp.src, and deposited, untouched, into the destination specified by gulp.dest. Anything that needs to happen to those files simply goes in as a step in the pipeline, and each file flows through that pipeline before proceeding to the next step in the pipeline. It’s the Unix architectural style of “pipes and filters,” brought to the Node.js ecosystem in an incredibly elegant way. Windows PowerShell is built on the same kind of architecture, for those who are thinking they’ve seen something like this before in the .NET universe.
Therefore, for example, if you just want to see Gulp touch each file through the pipeline, there’s a plug-in for that (“gulp-filelogger”), and it’ll print to the console on each file it touches:
gulp.task('copy-files', function () {
gulp.src(srcFiles)
.pipe(filelogger())
.pipe(gulp.dest(destDir));
});
This results in the following:
Teds-MacBook-Pro:gulpdemo tedneward$ gulp copy-files
[20:14:01] Using gulpfile ~/Projects/code/gulpdemo/gulpfile.js
[20:14:01] Starting 'copy-files'...
[20:14:01] Finished 'copy-files' after 14 ms
[20:14:01] [/Users/tedneward/Projects/code/gulpdemo/index.js]
Teds-MacBook-Pro:gulpdemo tedneward$
Notice how the output appears after Gulp reports it’s finished. Gulp can (and does) process these streams asynchronously much of the time, shortening the “build” times. Most of the time, developers neither know nor care that things are being done in parallel, but for those times that performing things in a precise sequence is important, not surprisingly, the Gulp plug-in community has some plug-ins that will serialize execution and make sure everything happens in sequence. Gulp 4.0 will add two new functions, parallel and serial, to make this more clear, but because it hasn’t shipped yet, you’ll have to wait on those.
By the way, Gulp itself consists purely of the four functions you’ve seen so far: gulp.task, gulp.watch, gulp.src and gulp.dest. Everything else is all plug-ins, npm modules or written by hand. This makes Gulp in and of itself extremely easy to understand. Depressingly easy, in fact, for article authors who are paid by the word.
Gulping a Lot All at Once
Gulp on its own is not all that complicated a tool, but as with any tool of this nature, its real strength lies in the vast array of plug-ins and complementary tools that have emerged out of the community around it. The full list is available at gulpjs.com/plugins, but Figure 2 shows a representative sample of a Gulp recipe, demonstrating how to automate the release of a project to GitHub, including the Git commands to push to master.
Figure 2 A Gulp Recipe
var gulp = require('gulp');
var runSequence = require('run-sequence');
var conventionalChangelog = require('gulp-conventional-changelog');
var conventionalGithubReleaser = require('conventional-github-releaser');
var bump = require('gulp-bump');
var gutil = require('gulp-util');
var git = require('gulp-git');
var fs = require('fs');
gulp.task('changelog', function () {
return gulp.src('CHANGELOG.md', {
buffer: false
})
.pipe(conventionalChangelog({
preset: 'angular' // Or to any other commit message convention you use.
}))
.pipe(gulp.dest('./'));
});
gulp.task('github-release', function(done) {
conventionalGithubReleaser({
type: "oauth",
token: '' // Change this to your own GitHub token.
}, {
preset: 'angular' // Or to any other commit message convention you use.
}, done);
});
gulp.task('bump-version', function () {
// Hardcode the version change type to "patch," but it might be a good
// idea to use minimist (bit.ly/2cyPhfa) to determine with a command
// argument whether you're doing a "major," "minor" or a "patch" change.
return gulp.src(['./bower.json', './package.json'])
.pipe(bump({type: "patch"}).on('error', gutil.log))
.pipe(gulp.dest('./'));
});
gulp.task('commit-changes', function () {
return gulp.src('.')
.pipe(git.add())
.pipe(git.commit('[Prerelease] Bumped version number'));
});
gulp.task('push-changes', function (cb) {
git.push('origin', 'master', cb);
});
gulp.task('create-new-tag', function (cb) {
var version = getPackageJsonVersion();
git.tag(version, 'Created Tag for version: ' + version, function (error) {
if (error) {
return cb(error);
}
git.push('origin', 'master', {args: '--tags'}, cb);
});
function getPackageJsonVersion () {
// Parse the json file instead of using require because require caches
// multiple calls so the version number won't be updated.
return JSON.parse(fs.readFileSync('./package.json', 'utf8')).version;
};
});
gulp.task('release', function (callback) {
runSequence(
'bump-version',
'changelog',
'commit-changes',
'push-changes',
'create-new-tag',
'github-release',
function (error) {
if (error) {
console.log(error.message);
} else {
console.log('RELEASE FINISHED SUCCESSFULLY');
}
callback(error);
});
});
This example demonstrates a number of things: how to run tasks in a particular sequence, using the Gulp plug-ins for generating a convention change set file, doing GitHub-style release messages, bumping the semantic version and a whole lot more. All from gulp release; that’s pretty powerful.
Wrapping Up
This hasn’t been a particularly code-heavy article, yet you just rebooted the entire application, gained a whole lot of functionality, and essentially brought the application up to the same level (and beyond) from what you’d been building for the past year or so. Gotta love scaffolding!
More important, having built all the parts piece by piece by hand prior to running the scaffolding, it’s much easier to understand the code as a whole and what’s happening where. For example, opening up routes.js will look familiar to the routing table you built by hand earlier, and the package.json (in the root of the project directory) will be bigger, but will remain the same as you’d been using.
The only new thing, in fact, beyond the use of Yeoman itself, is the introduction of a build tool to gather all the pertinent parts together into the right place, and that’ll be what I discuss next time. Until then … happy coding!
Ted Neward is a Seattle-based polytechnology consultant, speaker and mentor. He has written more than 100 articles, is an F #MVP, has authored and coauthored a dozen books. Reach him at ted@tedneward.com if you’re interested in having him come work with your team, or read his blog at blogs.tedneward.com.
Thanks to the following technical expert for reviewing this article: Shawn Wildermuth