Build what's next on GitHub, the place for anyone from anywhere to build anything.
Join us October 28-29 in San Francisco or online for GitHub Universe, our flagship developer event uniting people, agents, and the world's code.
We spend a lot of time optimizing the front end experience at GitHub. With that said, our asset (css, javascript, images) packaging and serving has evolved to be the best…
We spend a lot of time optimizing the front end experience at GitHub. With that said, our asset (css, javascript, images) packaging and serving has evolved to be the best setup I’ve seen out of any web application I’ve worked on in my life.
Originally, I was going to package what we have up into a plugin, but realized that much of our asset packaging is specific our particular app architecture and choice of deployment strategy. If you haven’t read up on our deployment recipe _read it now_. I cannot stress enough how awesome it is to have 14 second no downtime deploys. In any case, you can find the relevant asset bundling code in this gist
Our asset bundling is comprised of several different pieces:
Our file layout for CSS & JS is detailed in the README for Javascript, but roughly resembles something like this:
I like this layout because:
Some might say that relying on including everything is bad practice — but remember that web-based javascript is almost exclusively onDOMReady or later. That means that there is no dependency order problems. If you run into dependency order issues, you’re writing javascript wrong.
To help with this new bundle strategy, I’ve created some Rails helpers to replace your standard stylesheet_link_tag and javascript_include_tag. Because of the way we bundle files, it was necessary to use custom helpers. As an added benefit, these helpers are much more robust than the standard Rails helpers.
Here’s the code:
require 'find'
module BundleHelper
def bundle_files?
Rails.production? || Rails.staging? || params[:bundle] || cookies[:bundle] == "yes"
end
def javascript_bundle(*sources)
sources = sources.to_a
bundle_files? ? javascript_include_bundles(sources) : javascript_include_files(sources)
end
# This method assumes you have manually bundled js using a rake command
# or similar. So there better be bundle_* files.
def javascript_include_bundles(bundles)
output = ""
bundles.each do |bundle|
output << javascript_src_tag("bundle_#{bundle}", {}) + "n"
end
output
end
def javascript_include_files(bundles)
output = ""
bundles.each do |bundle|
files = recursive_file_list("public/javascripts/#{bundle}", ".js")
files.each do |file|
file = file.gsub('public/javascripts/', '')
output << javascript_src_tag(file, {}) + "n"
end
end
output
end
def javascript_dev(*sources)
output = ""
sources = sources.to_a
sources.each do |pair|
output << javascript_src_tag(Rails.development? ? "dev/#{pair[0]}" : pair[1], {})
end
output
end
def stylesheet_bundle(*sources)
sources = sources.to_a
bundle_files? ? stylesheet_include_bundles(sources) : stylesheet_include_files(sources)
end
# This method assumes you have manually bundled css using a rake command
# or similar. So there better be bundle_* files.
def stylesheet_include_bundles(bundles)
stylesheet_link_tag(bundles.collect{ |b| "bundle_#{b}"})
end
def stylesheet_include_files(bundles)
output = ""
bundles.each do |bundle|
files = recursive_file_list("public/stylesheets/#{bundle}", ".css")
files.each do |file|
file = file.gsub('public/stylesheets/', '')
output << stylesheet_link_tag(file)
end
end
output
end
def recursive_file_list(basedir, extname)
files = []
basedir = RAILS_ROOT + "/" + basedir
Find.find(basedir) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?.
Find.prune
else
next
end
end
files << path.gsub(RAILS_ROOT + '/', '') if File.extname(path) == extname
end
files.sort
end
end
require 'find'
module BundleHelper
def bundle_files?
Rails.production? || Rails.staging? || params[:bundle] || cookies[:bundle] == "yes"
end
def javascript_bundle(*sources)
sources = sources.to_a
bundle_files? ? javascript_include_bundles(sources) : javascript_include_files(sources)
end
# This method assumes you have manually bundled js using a rake command
# or similar. So there better be bundle_* files.
def javascript_include_bundles(bundles)
output = ""
bundles.each do |bundle|
output << javascript_src_tag("bundle_#{bundle}", {}) + "n"
end
output
end
def javascript_include_files(bundles)
output = ""
bundles.each do |bundle|
files = recursive_file_list("public/javascripts/#{bundle}", ".js")
files.each do |file|
file = file.gsub('public/javascripts/', '')
output << javascript_src_tag(file, {}) + "n"
end
end
output
end
def javascript_dev(*sources)
output = ""
sources = sources.to_a
sources.each do |pair|
output << javascript_src_tag(Rails.development? ? "dev/#{pair[0]}" : pair[1], {})
end
output
end
def stylesheet_bundle(*sources)
sources = sources.to_a
bundle_files? ? stylesheet_include_bundles(sources) : stylesheet_include_files(sources)
end
# This method assumes you have manually bundled css using a rake command
# or similar. So there better be bundle_* files.
def stylesheet_include_bundles(bundles)
stylesheet_link_tag(bundles.collect{ |b| "bundle_#{b}"})
end
def stylesheet_include_files(bundles)
output = ""
bundles.each do |bundle|
files = recursive_file_list("public/stylesheets/#{bundle}", ".css")
files.each do |file|
file = file.gsub('public/stylesheets/', '')
output << stylesheet_link_tag(file)
end
end
output
end
def recursive_file_list(basedir, extname)
files = []
basedir = RAILS_ROOT + "/" + basedir
Find.find(basedir) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?.
Find.prune
else
next
end
end
files << path.gsub(RAILS_ROOT + '/', '') if File.extname(path) == extname
end
files.sort
end
end
Our application.html.erb now looks something like this:
This includes jQuery and all javascript files under public/javascripts/common and public/javascripts/github (recursively). Super simple and we probably won’t need to change this for a very long time. We just add files to the relevant directories and they get included magically.
For pages that have heavy javascript load, you can still use the regular javascript_include_tag to include these files (we keep them under the public/javascripts/rogue directory).
The javascript_bundle and stylesheet_bundle helpers both assume that in production mode, there’ll be a corresponding bundle file. Since we are proactively generating these files, you need to create these manually on each deploy.
RAILS_ROOT ||= ENV["RAILS_ROOT"]
namespace :bundle do
task :all => [ :js, :css ]
task :js do
compression_method = "closure"
require 'lib/js_minimizer' if compression_method != "closure"
closure_path = RAILS_ROOT + '/lib/closure_compressor.jar'
paths = get_top_level_directories('/public/javascripts')
targets = []
paths.each do |bundle_directory|
bundle_name = bundle_directory.gsub(RAILS_ROOT + '/public/javascripts/', "")
files = recursive_file_list(bundle_directory, ".js")
next if files.empty? || bundle_name == 'dev'
target = RAILS_ROOT + "/public/javascripts/bundle_#{bundle_name}.js"
if compression_method == "closure"
`java -jar #{closure_path} --js #{files.join(" --js ")} --js_output_file #{target} 2> /dev/null`
else
File.open(target, 'w+') do |f|
f.puts JSMinimizer.minimize_files(*files)
end
end
targets << target
end
targets.each do |target|
puts "=> bundled js at #{target}"
end
end
task :css do
yuipath = RAILS_ROOT + '/lib/yuicompressor-2.4.1.jar'
paths = get_top_level_directories('/public/stylesheets')
targets = []
paths.each do |bundle_directory|
bundle_name = bundle_directory.gsub(RAILS_ROOT + '/public/stylesheets/', "")
files = recursive_file_list(bundle_directory, ".css")
next if files.empty? || bundle_name == 'dev'
bundle = ''
files.each do |file_path|
bundle << File.read(file_path) << "n"
end
target = RAILS_ROOT + "/public/stylesheets/bundle_#{bundle_name}.css"
rawpath = "/tmp/bundle_raw.css"
File.open(rawpath, 'w') { |f| f.write(bundle) }
`java -jar #{yuipath} --line-break 0 #{rawpath} -o #{target}`
targets << target
end
targets.each do |target|
puts "=> bundled css at #{target}"
end
end
require 'find'
def recursive_file_list(basedir, ext)
files = []
Find.find(basedir) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?. # Skip dot directories
Find.prune
else
next
end
end
files << path if File.extname(path) == ext
end
files.sort
end
def get_top_level_directories(base_path)
Dir.entries(RAILS_ROOT + base_path).collect do |path|
path = RAILS_ROOT + "#{base_path}/#{path}"
File.basename(path)[0] == ?. || !File.directory?(path) ? nil : path # not dot directories or files
end - [nil]
end
end
RAILS_ROOT ||= ENV["RAILS_ROOT"]
namespace :bundle do
task :all => [ :js, :css ]
task :js do
compression_method = "closure"
require 'lib/js_minimizer' if compression_method != "closure"
closure_path = RAILS_ROOT + '/lib/closure_compressor.jar'
paths = get_top_level_directories('/public/javascripts')
targets = []
paths.each do |bundle_directory|
bundle_name = bundle_directory.gsub(RAILS_ROOT + '/public/javascripts/', "")
files = recursive_file_list(bundle_directory, ".js")
next if files.empty? || bundle_name == 'dev'
target = RAILS_ROOT + "/public/javascripts/bundle_#{bundle_name}.js"
if compression_method == "closure"
`java -jar #{closure_path} --js #{files.join(" --js ")} --js_output_file #{target} 2> /dev/null`
else
File.open(target, 'w+') do |f|
f.puts JSMinimizer.minimize_files(*files)
end
end
targets << target
end
targets.each do |target|
puts "=> bundled js at #{target}"
end
end
task :css do
yuipath = RAILS_ROOT + '/lib/yuicompressor-2.4.1.jar'
paths = get_top_level_directories('/public/stylesheets')
targets = []
paths.each do |bundle_directory|
bundle_name = bundle_directory.gsub(RAILS_ROOT + '/public/stylesheets/', "")
files = recursive_file_list(bundle_directory, ".css")
next if files.empty? || bundle_name == 'dev'
bundle = ''
files.each do |file_path|
bundle << File.read(file_path) << "n"
end
target = RAILS_ROOT + "/public/stylesheets/bundle_#{bundle_name}.css"
rawpath = "/tmp/bundle_raw.css"
File.open(rawpath, 'w') { |f| f.write(bundle) }
`java -jar #{yuipath} --line-break 0 #{rawpath} -o #{target}`
targets << target
end
targets.each do |target|
puts "=> bundled css at #{target}"
end
end
require 'find'
def recursive_file_list(basedir, ext)
files = []
Find.find(basedir) do |path|
if FileTest.directory?(path)
if File.basename(path)[0] == ?. # Skip dot directories
Find.prune
else
next
end
end
files << path if File.extname(path) == ext
end
files.sort
end
def get_top_level_directories(base_path)
Dir.entries(RAILS_ROOT + base_path).collect do |path|
path = RAILS_ROOT + "#{base_path}/#{path}"
File.basename(path)[0] == ?. || !File.directory?(path) ? nil : path # not dot directories or files
end - [nil]
end
end
Throw this into lib/tasks/bundle.rake _and the corresponding YUI & Closure jars_ and then run rake bundle:all to generate your javascript. You can customize this to use the minifying package of your choice.
To make sure this gets run on deploy, you can add this to your deploy.rb:
namespace :deploy do
desc "Shrink and bundle js and css"
task :bundle, :roles => :web, :except => { :no_release => true } do
run "cd #{current_path}; RAILS_ROOT=#{current_path} rake bundle:all"
end
end
after "deploy:update_code", "deploy:bundle"
namespace :deploy do
desc "Shrink and bundle js and css"
task :bundle, :roles => :web, :except => { :no_release => true } do
run "cd #{current_path}; RAILS_ROOT=#{current_path} rake bundle:all"
end
end
after "deploy:update_code", "deploy:bundle"
The last step in optimizing your asset bundling for deploys is to tweak your production.rb config file to make asset serving a bit smarter. The relevant bits in our file are:
config.action_controller.asset_host = Proc.new do |source, request|
non_ssl_host = "http://assets#{source.hash % 4}.github.com"
ssl_host = "https://assets#{source.hash % 4}.github.com"
if request.ssl?
if source =~ /.js$/
ssl_host
elsif request.headers["USER_AGENT"] =~ /(Safari)/
non_ssl_host
else
ssl_host
end
else
non_ssl_host
end
end
repo = Grit::Repo.new(RAILS_ROOT)
js = repo.log('master', 'public/javascripts', :max_count => 1).first
css = repo.log('master', 'public/stylesheets', :max_count => 1).first
ENV['RAILS_ASSET_ID'] = js.committed_date > css.committed_date ? js.id : css.id
config.action_controller.asset_host = Proc.new do |source, request|
non_ssl_host = "http://assets#{source.hash % 4}.github.com"
ssl_host = "https://assets#{source.hash % 4}.github.com"
if request.ssl?
if source =~ /.js$/
ssl_host
elsif request.headers["USER_AGENT"] =~ /(Safari)/
non_ssl_host
else
ssl_host
end
else
non_ssl_host
end
end
repo = Grit::Repo.new(RAILS_ROOT)
js = repo.log('master', 'public/javascripts', :max_count => 1).first
css = repo.log('master', 'public/stylesheets', :max_count => 1).first
ENV['RAILS_ASSET_ID'] = js.committed_date > css.committed_date ? js.id : css.id
There’s three important things going on here.
First— If you hit a page using SSL, we serve all assets through SSL. If you’re on Safari, we send all CSS & images non-ssl since Safari doesn’t have a mixed content warning.
It is of note that many people suggest serving CSS & images non-ssl to Firefox. This was good practice when Firefox 2.0 was standard, but now that Firefox 3.0 is standard (and obeys cache-control:public as it should) there is no need for this hack. Firefox does have a mixed content warning (albeit not as prominent as IE), so I choose to use SSL.
Second— We’re serving assets out of 4 different servers. This fakes browsers into downloading things faster and is generally good practice.
Third— We’re hitting the git repo on the server (note our deployment setup) and getting a sha of the last changes to the public/stylesheets and public/javascripts directory. We use that sha as the ASSET_ID (the bit that gets tacked on after css/js files as ?sha-here).
This means that if we deploy a change that only affects app/application.rb we don’t interrupt our user’s cache of the javascripts and stylesheets.
What all of this adds up to is that our deploys have almost no frontend consequence unless they intend to (changing css/js). This is huge for a site that does dozens of deploys a day. All browser caches remain the same and there isn’t any downtime while we bundle up assets. It also means we’re not afraid to deploy changes that may only affect one line of code and some minor feature.
All of this is not to say there isn’t room for improvement in our stack. I’m still tracking down some SSL bugs, and always trying to cut down on the total CSS, javascript and image load we deliver on every page.