replacing gitolite with a redis-backed git server

Pogoapp has started growing faster the past couple months, and we've started hitting some roadbumps as we scale up. We thought it'd be worth doing some occasional blogging about specific situations we've run into and how we've dealt with them to keep the platform response as we grow.

Our git repository management has been a particular pain point for a while, but as we've grown towards 300 repositories the wheels really started to come off. We'd been managing the repositories and config with an internal system built for automating gitolite. Gitolite is one of the more popular open-source tools for administering a git server, and almost all the configuration for the system itself is done in the special gitolite-admin repository.

So, any time a user ran a command requiring a change to the git server (e.g. adding a new repo/user/ssh key), a background ruby worker would try to clone/update a local working directory of the gitolite-admin repo, regenerate the config file from our database records, create/update any user key files, and then commit and push the changes back to the server, where a gitolite "post-update" hook parses the config and setups up all the SSH keys in the git user's ~/.ssh/authorized_keys file

The whole thing was quite a duct-taped mess, creaking and whirring and breaking fairly often.

We figured other people have to have had this problem before, and we went looking for solutions. After a lot of digging around, the best clues we came up with were the gitorious source code, which includes a database-backed ruby git-daemon, and especially this post a few years back from Github:

We have patched our SSH daemon to perform public key lookups from our MySQL database. Your key identifies your GitHub user and this information is sent along with the original command and arguments to our proprietary script called Gerve (Git sERVE). Think of Gerve as a super smart version of git-shell.

This led us to hunt down this stackoverflow post which includes a great little find, a link to a github repository with an OpenSSH "script auth" patch (which has useful forks). Patching OpenSSH is pretty high on the list of "insane sysadmin ideas", and mizzy's fork includes the fantastic "Security" section, which begins: "Are you kidding? We forked a fork of OpenSSH. Though we only added a few dozen lines, odds are we also incorporated a few hundred buffer overflow vectors."

So, caveat emptor. Luckily we're going to be running this in an isolated LXC container and only allowing the git user access.

Working off the example from that repo, we patched together some less-than-beautiful code that seems to get the job done for OpenSSH/redis auth:

key = STDIN.gets.strip
user = ARGV.first

unless user == 'git'
  log "[ERROR] invalid user!"
  Kernel.exit(1)
end

unless key =~ /^ssh-(?:dss|rsa) [A-Za-z0-9+\/](=+)?+/
  log("[ERROR] invalid key!") if LOGGING
  Kernel.exit(1)
end

require 'redis'
uri = URI.parse(ENV['REDISDB_URL'])
uri.port ||= 6379
redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password, :db => 4)

user_val = redis.get(key)
user_id, name = user_val.split(':') if user_val

if user_id and name
  STDOUT.print %Q[command="bundle exec /app/pogo-git #{user_id} #{name}",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty]
  Kernel.exit(0)
else
  log "lookup failed: #{key} | #{user_val} | #{user_id} | #{name}"
  Kernel.exit(1)
end

Assuming your key is found in the database, the git SSH command gets passed onto a second script, our redis-backed git/gitolite-shell:

require 'redis'

class GitAction
  attr_reader :user_id, :repo, :cmd
  def initialize(user_id, ssh_command)
    @user_id = user_id
    
    # input e.g. "git-upload-pack '/test.git'" or "git-upload-pack 'test.git'"
    @cmd, repo_str = ssh_command.split(' ', 2)
    repo_fname = File.basename(repo_str).tr("'", '')
    
    unless cmd.match(/^git[ -]upload-pack$/) || cmd.match(/^git[ -]receive-pack$/)
      $stderr.puts "Only git commands are allowed, not: #{cmd}"
      Process.exit(2)
    end
    
    # FIXME: junky regex
    unless repo_fname.match(/\w+\.git/)
      $stderr.puts "Repository parameter incorrect: #{repo_fname}"
      Process.exit(2)
    end
    
    @repo = File.basename(repo_fname, '.git')
    
    ensure_setup if valid_user?
    true
  end
  
  def valid_user?
    repo_user_ids.include?(user_id)
  end
  
  # TODO: read-only users?
  def allowed?
    valid_user?
  end
  
  def repo_user_ids
    @repo_user_ids ||= redis.smembers(repo) || []
  end
  
  def ensure_setup
    `GIT_TEMPLATE_DIR=/app/repo-template git init --bare #{path} &> /dev/null` unless Dir.exists?(path)
    `ln -nfs /app/hooks #{path}/hooks`
  end
  
  def repo_dir
    '/app/share/gitolite/repositories'
  end
  
  def path
    "#{repo_dir}/#{repo}.git"
  end
  
  def redis
    @redis ||= begin
      uri = URI.parse(ENV['REDISDB_URL'])
      uri.port ||= 6379
      redis = Redis.new(:host => uri.host, :port => uri.port, :password => uri.password, :db => 4)
    end
  end
end

###

unless ssh_command = ENV["SSH_ORIGINAL_COMMAND"]
  $stderr.puts "SSH not allowed to the git account."
  Process.exit(1)
end

user_id, name = *ARGV

begin
  git_action = GitAction.new(user_id, ssh_command)
  
  if git_action.allowed?
    ENV['GL_REPO'] = git_action.repo
    ENV['GL_USER'] = name
    
    exec "/usr/bin/#{git_action.cmd} #{git_action.path}"
    Process.exit(0)
  else
    $stderr.puts "#{name} does not have permission to #{git_action.cmd}"
    Process.exit(5)
  end
rescue => e
  $stderr.puts "Error!"
  Process.exit(4)
end

This command parses the name of the repository you're trying to access and what command you're trying to run, validates it all looks ok and that your user is allowed access to the specific repository, and if so, exec's the requested command. We also setup a bare, empty repository for you if needed, and symlink in hooks from a central location.

We rolled this live a little over a week ago and its been a huge improvement. Any current Pogoapp users who need private git hosting for can add new repositories to their app with a simple:

pogo addons:add git:dev --name my-repo-name

and you'll get a new repository:

git@git.pogoapp.com:my-repo-name.git