Yuki & Moto Press

Hoe Developer's Guide - Build, Package and Publish Gems with Rake Tasks - Ready-to-Use Build Scripts (Book Edition)

by Ryan Davis, et al

Contents

Notes

This is the original source reformatted in a single-page book edition (using the Manuscripts format).

See the source repo for how the book gets auto-built with "plain" Jekyll - of course - and hosted on GitHub Pages.

Onwards.

Hoe Developer's Guide

Contents

Build, Package and Publish Gems with Rake Tasks - Ready-to-Use Build Scripts

What is Hoe?

Hoe is a library that provides extensions to rake to automate every step of the development process from genesis to release. It provides project creation, configuration, and a multitude of tasks including project maintenance, testing, analysis, and release. We found rake to be an incredible vehicle for functionality in the abstract, but decidedly lacking in concrete functionality. We filled in all the blanks we could through a “Hoe-spec”:

require "hoe"
 Hoe.spec "project_name" do
 developer "Ryan Davis", "ryand-ruby@zenspider.com"
 # ...
end

A Hoe-spec declares everything about your project that is different from the defaults. From that, Hoe creates a multitude of tasks and a gemspec used for packaging and release. Updating Hoe updates all your projects that use Hoe. That’s it. Nothing more is needed. Everything is DRY (Don’t Repeat Yourself).

A Brief History of Hoe

Hoe was extracted from pain, not from fun. It was decidedly not written in a vacuum. Pain to me is repetition and mindless/needless work. At the time of this writing, a subset of seattle.rb members have 88 projects with 439 releases; all but 86 of those releases were done with Hoe. Pre-Hoe, an inordinate amount of time and effort was put into keeping those projects in sync with each other. In other words, we know what we’re talking about here and it ain’t pretty.

Every time we found a new task that was useful to one project, it was probably useful to the rest of them… but used in a slightly different way. Resolving those edits across all the projects took time away from writing fun/good/useful code. Every time we found a snafu in our release process and wanted to improve it, we had to propagate those changes lest we have another snafu. In short, we had code duplication across our projects, but on the release/package/process side. We weren’t DRY and at the time, there wasn’t much available for process-oriented libraries. Through this pain, Hoe was born.

Why Use Hoe?

Projects, the DRY way

dry drī , adjective
  1. A software engineering principal stating “Every piece of knowledge must have a single, unambiguous, authoritative representation within a system”.

Hoe focuses on keeping everything in its place in a useful form and intelligently extracting what it needs. There is no duplication of this information across your project. As a result, there are no extra YAML files, config directories, ruby files, or any other artifacts in your release that you wouldn’t already have. As such, there is a lot less extra work you need to do to maintain your projects.

Update all your projects in 1 easy step

% gem update hoe

Hoe does minimal code generation. Just the barest amounts of Hoe are exposed in your project (just the Rakefile). As such, updating Hoe’s gem is an automatic update for all your projects. The worst upgrades of Hoe require changing a line or two in your Rakefile.

Change Propagation

Code Generation -versus- Code Reuse

With code generation, each new project adds N new dependencies across all existing projects. Any change to any project might propagate to every other project. With code reuse, adding a new project has no impact on any other project.

New projects in 1 easy step

% sow project_name

That’s all you need to create a new project. And you can customize that to your heart’s content. All the files used for project creation are tweakable and are located in ~/.hoe_template.

Run your tests in 1 easy step

% rake

Everything else is taken care of. Period.

Releasing in 1 easy step

% rake release VERSION=x.y.z

That really is all there is to it. As-is, that command cleans your project, packages your project into a gem and uploads it to rubygems.org. Tweak a couple knobs and add a plugin or two and you can have it running your tests, validating against the manifest, posting to your blog or mailing lists, or anything else you can think of.

That VERSION=x.y.z is there as a last-chance sanity check that you know what you’re releasing. You’d be surprised how blurry eyed/brained you get at 3AM. This check helps a lot more often than it should.

Noticing a pattern yet?

Hoe is designed to reduce code in your project and deliver a lot of power to you across all your projects. It takes the mundane and redundant out of your project and lets you focus the actual project.

Extensibility

Hoe.plugin :my_hoe_extension

Hoe has a powerful plugin system that lets you tweak to your heart’s content. Want to have git integration? No problem! There is a plugin for that or whatever VC you like. Want to automate sending out an email or posting to your blog every time you release? There is a plugin for that too. Writing a C extension and you don’t want to deal with all the hassle of getting it set up and built? Done.

And if you have something specific to your company or used across all of your open source projects, plugin writing is a great way to DRY up your projects. Imagine taking all those tasks/*.rake files and substituting them with a simple Hoe.plugin line. All that code can be consolidated, put into its own gem, versioned properly, released, and reused with ease. No more version skew. No more copy and paste. No more junk littering your projects.

This is the real power of Hoe.

Creating a new Project

From Scratch

The easiest way to get started with Hoe is to use its included command-line tool sow:

% sow my_shiny_project

That will create a new directory my_shiny_project with a skeletal project inside. You need to edit the Rakefile with developer information in order to meet the minimum requirements of a working Hoe-spec. You should also go fix all the things it points out as being labeled with FIX in the README.txt file.

require 'hoe'

 Hoe.spec 'my_shiny_project' do
   developer 'Ryan Davis', 'ryand-ruby@zenspider.com'

   extra_deps << 'whatevs'
 end

Using Sow Templates

If you’re planning on releasing a lot (aka: 2 or more) of packages and you’ve got certain recipes you like to have in your project, do note that sow uses a template directory and ERB to create your project. The first time you run sow it creates ~/.hoe_template. Make modifications there and every subsequent project will have those changes. For example, my default Hoe template looks like:

Hoe.plugin :isolate
Hoe.plugin :seattlerb

Hoe.spec "<%= project %>" do
  developer "Ryan Davis", "ryand-ruby@zenspider.com"
  license "MIT"
end

Work the Way You Want to Work

Hoe tries its best to stay out of your way. While it follows a bunch of conventions, it doesn’t enforce very much at all.

Version control agnostic.

Hoe doesn’t assume anything about HOW you work. Git? SVN? Perforce? Great! Hoe doesn’t care, but there is probably a hoe plugin that will support whatever you use.

Test framework agnostic.

Hoe doesn’t care how you test your code. Hoe works out of the box for test/unit, minitest, shoulda, rspec… And it is very easy to support others.

Project Structure

Hoe encourages the canonical rubygems setup with all the usual extras to make your project maintainable and approachable:

lib/\*\*/\*.rb

All your source goes in here, as usual.

{test,spec}/\*\*/\*.rb

All your tests go in here, as usual.

bin/\*

Commandline executables go in here, if you have any.

README.txt

Most projects have a readme file of some kind that describes the project. Hoe is no different. The readme file points the reader towards all the information they need to know to get started including a description, relevant urls, code synopsis, license, etc. Hoe knows how to read a basic rdoc/markdown formatted file to pull out the description (and summary by extension), urls, and extra paragraphs of info you may want to provide in news/blog posts.

History.txt

Every project should have a document describing changes over time. Hoe can read this file (also in rdoc/markdown) and include the latest changes in your announcements.

Manifest.txt

Every project should know what it is shipping. This is done via an explicit list of everything that goes out in a release. Hoe uses this during packaging so that nothing wrong or embarrassing is picked up.

VERSION

Releases have versions and I’ve found it best for the version to be part of the code. You can use this during runtime in a multitude of ways. Hoe finds your VERSION constant in your code and uses it automatically during packaging.

class MyShinyProject
  VERSION = "1.0.0"
  # ...
end

Extending Hoe with Plugins

Hoe has a flexible plugin system with the release of 2.0. This allowed Hoe to be refactored. That in and of itself was worth the effort. Probably more important is that it allows you to customize your projects’ tasks in a modular and reusable way

Using Hoe Plugins

Using a Hoe plugin is incredibly easy. Activate it by calling Hoe.plugin like so:

Hoe.plugin :minitest

This will activate the Hoe::Minitest plugin, attach it and load its tasks and methods into your Hoe-spec. Easy-peasy!

Here are some examples of plugins that are available:

hoe-bundler gem

Generates a Gemfile based on a Hoe’s declared dependencies.

hoe-debugging gem, :compiler plugin (ships with hoe)

Help you build and debug your ruby C extensions.

hoe-doofus gem

Helps keep you from messing up gem releases.

hoe-git & hoe-hg

These plugins provide git and mercurial integration.

hoe-seattlerb

Plugin bundle for minitest, email announcements, perforce support, and release branching.

Writing Hoe Plugins

If the existing plugins don’t meet your needs, it is very simple to write your own. A plugin can be as simple as:

module Hoe::CompanysStuff
 attr_accessor :thingy

 def initialize_companys_stuff # optional
   self.thingy = 42
 end

 def define_companys_stuff_tasks # required
   task :thingy do
     puts thingy
   end
 end
end

Not terribly useful, but you get the idea. This example exercises both plugin methods (initialize_#{plugin} and define_#{plugin}_tasks) and adds an accessor method to the Hoe instance. Only the define method is required but sometimes it is left blank if all you want is an initialize method that sets some values for you.

How Plugins Work

Hoe plugins are made to be as simple as possible, but no simpler. They are modules defined in the Hoe namespace and have only one required method (define_#{plugin}_tasks) and one optional method (initialize_#{plugin}). Plugins can also define their own methods and they’ll be available as instance methods to your Hoe-spec. Plugins have 4 simple phases:

Loading

When Hoe is loaded the last thing it does is to ask rubygems for all of its plugins. Plugins are found by finding all files matching "hoe/\*.rb" via installed gems or $LOAD_PATH. All found files are then loaded.

Activation

All of the plugins that ship with Hoe are activated by default. This is because they’re providing the same functionality that the previous Hoe was and without them, it’d be mostly useless. Other plugins are activated by:

Hoe.plugin :thingy

Put this above your Hoe-spec. All it does is add :thingy to the array returned by Hoe.plugins. You could also deactivate a plugin by removing it from Hoe.plugins although that shouldn’t be necessary for the most part.

One nice thing about plugins is that they are “soft”. If you download a project from github and it requests a plugin that you don’t have installed, rake won’t freak out and die. You may not get the full set of tasks that the project developers have available, but you still have enough to play with the project.

Initialization

When your Hoe-spec is instantiated, it calls extend on itself with all known plugin modules. This adds the method bodies to the Hoe-spec instance and allows for the plugin to work as part of the spec itself. Once that is over, activated plugins have their optional define initialize_#{plugin} methods called. This lets them set needed instance variables to default values. Finally, the Hoe-spec block is evaluated so that project specific values can override the defaults.

Task Definition

Finally, once the user’s Hoe-spec has been evaluated, all activated plugins have their define_#{plugin}_tasks method called. This method must be defined and it is here that you’ll define all your tasks.

Questions & Counterpoints

“Why should I maintain a Manifest.txt when I can just write a glob?”

manifest2 ˈmønəˈfɛst   ˈmanɪfɛst , noun

a document giving comprehensive details of a ship and its cargo and other contents, passengers, and crew for the use of customs officers.

Imagine, you’re a customs inspector at the Los Angeles Port, the world’s largest import/export port. A large ship filled to the brim pulls up to the pier ready for inspection. You walk up to the captain and his crew and ask “what is the contents of this fine ship today” and the captain answers “oh… whatever is inside”. The mind boggles. There is no way in the world that a professionally run ship would ever run this way and there is no way that you should either.

Professional software releases know exactly what is in them, amateur releases do not. “Write better globs” is the response I often hear. I consider myself and the people I work with to be rather smart people and if we get them wrong, chances are you will too. How many times have you peered under the covers and seen .DS_Store, emacs backup~ files, vim swp files and other files completely unrelated to the package? I have far more times than I’d like.

We’ve even seen a gem that includes every gem released before it inside (recursively!).

You avoid all of this pain and embarrassment with a simple text file.

“Why not just write gemspecs?”

I’ve done that and it is way too much work.

First off, the question is short-sighted. A project is a lot more than just a gemspec and Hoe handles all of it. Second, it isn’t DRY. All my projects have a history file, a readme, some code with a version string, etc. Why should I duplicate all of that information into the gem spec when I can have code do it for me automatically? It is less error prone as a result. I screw up things, Hoe doesn’t.

See that Hoe spec above for the fictional “my_shiny_project” project? This is the corresponding gem spec in all its glory (as cleaned up as I can/am willing to get it):

# -*- encoding: utf-8 -*-

Gem::Specification.new do |s|
  s.name             = "my_shiny_project"
  s.version          = "1.0.0"

  s.authors          = ["Ryan Davis"]
  s.description      = "..."
  s.email            = ["ryand-ruby@zenspider.com"]
  s.executables      = ["my_shiny_project"]
  s.extra_rdoc_files = [...]
  s.files            = [...]
  s.homepage         = "..."
  s.rdoc_options     = ["--main", "README.txt"]
  s.summary          = "..."
  s.test_files       = [...]

  s.cert_chain       = ["/Users/ryan/.gem/gempublic_cert.pem"]
  s.signing_key      = "/Users/ryan/.gem/gem-private_key.pem"

  s.add_runtime_dependency(%q<whatevs>, [">= 0"])
end

And that doesn’t even cover the rake tasks… the code duplication… the versioning… etc.

Gross, no? If you say “no”, well, have fun with that. I won’t try to convince you any further. I’ve got real stuff to do while you tweak your spec ad nauseum.

“What about (newgem|bones|echoe|joe|gemify|…)?”

Smoke ‘em if ya got ‘em.

All I can really say is that Hoe works really well for me and a lot of others (rdoc, nokogiri, etc).

Hello, Hoe! - Let's Build a Gem Together!

Contents

Note: This chapter was written by Gerald Bauer. It’s taken from the book Gem Series ++ Project Automation & Database Documentation Tools.

What’s Hoe? - Automate, Automate, Automate

Hoe is a library that bundles ready-to-use rake tasks to help you build, package and publish your own gems. Thanks to Ryan Davis zenspider and friends (from Seattle.rb) for polishing the gem all those years - more than 100+ releases - leading to today’s version 3.16.

Let’s create a bare bones gem (hellohoe) and publish it on RubyGems.org.

Set up your gem by adding a build script, readme, change log and manifest

To use Hoe together with your own code use the following structure:

/hellohoe
  + README.txt        - Description in plain text
  + History.txt       - Version change log in plain text
  + Manifest.txt      - List of all files to include in plain text
  + Rakefile          - Build script (requires your name and email)  
  + /lib
     + hellohoe.rb    - Ruby code to bundle up into gem here

Note: You can grab all files from this post from the hellohoe GitHub repo.

Let’s look at hellohoe.rb:

class HelloHoe
  VERSION = '0.1.0'

  # your code here
end

Hoe requires a VERSION string in your Ruby code that you can reference in your build script. Let’s look at the build script, that is, Rakefile next:

require 'hoe'                # pull in the hoe machinery (that is, ready-to-use rake tasks)
require './lib/hellohoe.rb'

Hoe.spec 'hellohoe' do
  self.version = HelloHoe::VERSION

  self.author  = '[Your Name Here]'
  self.email   = '[Your Email Here]'

  # or use shortcut
  # self.developer( '[Your Name Here]', '[Your Email Here]' )
end

As a minimum Hoe requires you to set the author and email fields in the gemspec. As a shortcut you can use the developer method to set it all at once.

Next Hoe requires a readme in plain text stored in README.txt:

= hellohoe

* https://github.com/planetruby/hellohoe

== DESCRIPTION:

Sample on how to use Hoe Rake tasks to build, package and publish gems.

== LICENSE:

The hellohoe sources are dedicated to the public domain.

Hoe will use the link from the first section, that is, github.com/planetruby/hellohoe to auto-fill the homepage field in the gemspec and will use the description to auto-fill the summary field and the description in the gemspec.

Next Hoe requires a version changelog in plain text stored in History.txt:

=== 0.1.0 / 2018-01-08

* Everything is new. First release.

Hoe will use the changelog to auto-fill the changes field in the gemspec and use the changelog for emails and announcements.

Finally, Hoe requires a manifest - a list of all files to include in plain text stored in Manifest.txt:

History.txt
Manifest.txt
README.txt
Rakefile
lib/hellohoe.rb

Now you’re all set to use Hoe’s rake tasks to build, package and publish gems and more. You can list all tasks by running rake -T. Resulting in:

rake announce              # publish   # Announce your release.
rake audit                 # test      # Run ZenTest against the package.
rake check_extra_deps      # deps      # Install missing dependencies.
rake check_manifest        # debug     # Verify the manifest.
rake clean                 # clean     # Clean up all the extras.
rake clobber_docs          # publish   # Remove RDoc files
rake clobber_package       # package   # Remove package products
rake config_hoe            # debug     # Create a fresh ~/.hoerc file.
rake dcov                  # publish   # Generate rdoc coverage report
rake debug_email           # publish   # Generate email announcement file.
rake debug_gem             # debug     # Show information about the gem.
rake default               # test      # Run the default task(s).
rake deps:email            # deps      # Print a contact list for gems dependent on this gem
rake deps:fetch            # deps      # Fetch all the dependent gems of this gem into tarballs
rake deps:list             # deps      # List all the dependent gems of this gem
rake docs                  # publish   # Generate rdoc
rake gem                   # package   # Build the gem file hellohoe-0.1.gem
rake generate_key          # signing   # Generate a key for signing your gems.
rake install_gem           # package   # Install the package as a gem.
rake install_plugins       # deps      # Install missing plugins.
rake newb                  # newb      # Install deps, generate docs, run tests/specs.
rake package               # package   # Build all the packages
rake post_blog             # publish   # Post announcement to blog.
rake publish_docs          # publish   # Publish RDoc to wherever you want.
rake release               # package   # Package and upload; Requires VERSION=x.y.z (optional PRE=a.1)
rake release_sanity        # package   # Sanity checks for release
rake release_to_gemcutter  # gemcutter # Push gem to gemcutter.
rake repackage             # package   # Force a rebuild of the package files
rake ridocs                # publish   # Generate ri locally for testing.

Using debug_gem, gem, package, install_gem tasks

Let’s try some Hoe tasks. Run rake debug_gem to show the gemspec Hoe generates from your build script settings, readme, change log and manifest. Next, let’s build the gem. Run rake gem. Resulting in:

mkdir -p pkg
  Successfully built RubyGem
  Name: hellohoe
  Version: 0.1.0
  File: hellohoe-0.1.0.gem
mv hellohoe-0.1.0.gem pkg/hellohoe-0.1.0.gem

Hoe will place your gem in the pkg folder. If you run rake package Hoe will bundle up all your sources in a tar’ed and gzipped package (e.g. pkg/hellohoe-0.1.0.tgz).

Next, let’s test drive the gem. Run rake install_gem to install the gem and try it in the Ruby console:

$ irb
>> require 'hellohoe'
=> true
>> HelloHoe::VERSION
=> "0.1.0"

Checking and updating your manifest with check_manifest

Hoe includes a check_manifest task that lets you check the manifest against your files and see if any files are missing or need to get added.

If you run the task the first time you need to create a ~/.hoerc setting file first that includes a regex (regular expression) pattern that excludes files from the manifest check. To create a new ~/.hoerc file run rake config_hoe. Resulting in a file such as:

---
exclude: !ruby/regexp /tmp$|CVS|TAGS|\.(svn|git|DS_Store)/
signing_key_file: ~/.gem/gem-private_key.pem
signing_cert_file: ~/.gem/gem-public_cert.pem
publish_on_announce: true
blogs:
- user: user
  password: password
  url: url
  blog_id: blog_id
  extra_headers:
    mt_convert_breaks: markdown

Now let’s try rake check_manifest. If everything is in order (no files missing or waiting to get added). You will see:

rm -r doc
rm -r pkg
rm Manifest.tmp

Let’s create a new Todo.txt file and let’s retry rake check_manifest. Now you will see a diff:

@@ -2,4 +2,5 @@
 Manifest.txt
 README.txt
 Rakefile
+Todo.txt
 lib/hellohoe.rb

Using the release task to upload (push) your gem to RubyGems.org

Next, let’s upload (push) the gem to RubyGems.org using the release task.

Before you can upload to RubyGems.org you will need to setup an account and save your RubyGems.org API key on your computer. Issue the command to store your RubyGems.org API key on your computer (only needed the first time):

$ curl -u carlos https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials

Now you’re ready to use hoe to upload (push) your gem to RubyGems.org without requiring to enter a user and password. Run the command rake release and pass along the required release version. Example:

$ rake release VERSION=0.1.0

Check your RubyGems.org project page (e.g. rubygems.org/gems/hellohoe) if it all worked. Note, that it will take a minute or more until your uploaded gem gets added to the public RubyGems index. You can check if your gem is available using the list command with the -r (remote) switch. Example:

$ gem list hellohoe -r

*** REMOTE GEMS ***

hellohoe (0.1.0)

That’s it.

Appendix - Hello, Hoe! Sample Sources

Contents

Directory Structure

Full source code listings for the hellohoe gem:

|   README.txt
|   History.txt
|   Manifest.txt
|   Rakefile
|
\---lib/
      hellohoe.rb

README.txt

= hellohoe

* https://github.com/planetruby/hellohoe

== DESCRIPTION:

Sample on how to use Hoe Rake tasks to build, package and publish gems.

== LICENSE:

The hellohoe sources are dedicated to the public domain. Use it as you please with no restrictions whatsoever.

History.txt

=== 0.1 / 2018-03-05

* Everything is new. First release.

Manifest.txt

History.txt
Manifest.txt
README.txt
Rakefile
lib/hellohoe.rb

Rakefile

require 'hoe'
require './lib/hellohoe.rb'

Hoe.spec 'hellohoe' do
  self.version = HelloHoe::VERSION

  self.author  = 'Gerald Bauer'
  self.email   = 'gerald.bauer@example.com'
end

lib/hellohoe.rb

class HelloHoe

  VERSION = '0.3'

  # your code here
end
Fork me on GitHub