Ingot

Ingot is an attempt at Git itself in Common Lisp. It reads and writes the repositories that git makes, byte for byte: loose and packed objects, deltas included, refs and packed-refs, trees, commits and tags. On them it does what the porcelain does: the log, a file at a commit, diffs, branches, three-way merges and merge bases, archives of a tree; and it speaks Git's smart protocol over HTTP on both sides, so a git clone from it and a git push to it work, and so does a fetch by it from another server. No subprocess and no git binary anywhere, and every object id is the one git would compute.

It is early, and in the open. The source is at https://gitlab.common-lisp.net/clo/ingot, where issues and merge requests are welcome; it was written as the Git engine of Anvil, a forge in Common Lisp.

Loading it

Ingot is a clone and two forms away.

From source, in any Lisp with ASDF and Quicklisp (it is known to load on SBCL and Clozure CL; its only dependencies are chipz, salza2, babel and bordeaux-threads):

$ git clone https://gitlab.common-lisp.net/clo/ingot.git

(asdf:load-asd "/path/to/ingot/ingot.asd")
(ql:quickload :ingot)

The session, a step at a time

What follows is the session the front page turns through, run as written in Clozure CL. The answers are what it answered. The commit ids carry the clock, so your own run differs there; the blob's id at the end does not, because the bytes are the same.

1. git init --bare notes.git

A bare repository in the current directory, as git init --bare would make it: HEAD, config, objects/, refs/. The identity is the name and address a commit carries; the time is taken when the commit is made.

CL-USER> (ql:quickload "ingot")
("ingot")
CL-USER> (defparameter *repo*
           (ingot:init-repository "notes.git/"))
*REPO*
CL-USER> (defparameter *me*
           (ingot:make-identity "A Visitor"
                                "you@example.org"))
*ME*

2. git add README.md lisp/hello.lisp; git commit -m 'First notes'

A commit is the base tree plus a list of actions: :create, :update, :delete and :move, each with a path. There is no work tree and no staging; the tree is built from the actions and written, then the commit, then the branch moves. The answer is the commit's id.

CL-USER> (ingot:make-commit! *repo* :branch "master"
           :author *me* :message "First notes"
           :actions
           '((:action :create :path "README.md"
              :content "# Notes, kept with Ingot")
             (:action :create :path "lisp/hello.lisp"
              :content "(defun hello () 'hello)")))
"aa75a5b51eca0aea9485c445bdace69bdaaa9aa5"

3. git commit -am 'Hello says more'

The second commit updates one file. Its parent is the branch's tip, found by the branch name; the tree it starts from is the tip's.

CL-USER> (ingot:make-commit! *repo* :branch "master"
           :author *me* :message "Hello says more"
           :actions
           '((:action :update :path "lisp/hello.lisp"
              :content "(defun hello () '(hello world))")))
"9860f1e2cbfe0ac500580e0ffa266864f40308c9"

4. git log --oneline

log-commits walks the history newest first from any commit, with :limit and a :path filter when wanted. Each commit answers its subject, message, author, committer and parents.

CL-USER> (defparameter *head* (ingot:head-oid *repo*))
*HEAD*
CL-USER> (mapcar #'ingot:commit-subject
                 (ingot:log-commits *repo* *head*))
("Hello says more" "First notes")

5. git ls-tree -r HEAD

tree-paths walks a tree recursively and answers every path with its entry (mode, name, id). The commit's tree comes from the commit.

CL-USER> (mapcar #'car
          (ingot:tree-paths *repo*
           (ingot:commit-tree
            (ingot:read-commit *repo* *head*))))
("README.md" "lisp/hello.lisp")

6. git show HEAD:lisp/hello.lisp

blob-at finds a path in the tree of a commit and answers the blob's bytes; octets-to-string reads them as UTF-8.

CL-USER> (ingot:octets-to-string
          (ingot:blob-at *repo* *head* "lisp/hello.lisp"))
"(defun hello () '(hello world))"

7. git diff HEAD~1 HEAD -- lisp/hello.lisp

A commit's parents are ids. The diff here is a Myers line diff of the two texts, printed as git diff prints it; commit-diff answers the changed paths of a commit, and tree-diff compares two trees.

CL-USER> (defparameter *parent*
           (first (ingot:commit-parents
                   (ingot:read-commit *repo* *head*))))
*PARENT*
CL-USER> (ingot:unified-diff
          (ingot:octets-to-string
           (ingot:blob-at *repo* *parent* "lisp/hello.lisp"))
          (ingot:octets-to-string
           (ingot:blob-at *repo* *head* "lisp/hello.lisp")))
"--- a
+++ b
@@ -1 +1 @@
-(defun hello () 'hello)
+(defun hello () '(hello world))
"

8. git checkout -b topic master; git commit; git merge topic

A branch that does not exist yet starts from :start-branch. The merge is a fast-forward, since master has not moved: it answers the topic's own commit, and both branches point at it. When both sides have moved, the merge is a three-way one with a merge commit, and a conflict stops it, saying where.

CL-USER> (ingot:make-commit! *repo* :branch "topic"
           :start-branch "master" :author *me*
           :message "A to-do list"
           :actions '((:action :create :path "TODO.md"
                       :content "- read the Ingot page")))
"6c8217f9ee00b74bfe52576ead9b9f38f25295d4"
CL-USER> (ingot:merge-branches! *repo* "topic" "master"
                                :author *me*)
"6c8217f9ee00b74bfe52576ead9b9f38f25295d4"
CL-USER> (ingot:branches *repo*)
(("master" . "6c8217f9ee00b74bfe52576ead9b9f38f25295d4")
 ("topic" . "6c8217f9ee00b74bfe52576ead9b9f38f25295d4"))

9. echo -n hello | git hash-object --stdin

The id git gives a blob is the SHA-1 of blob <length>\0 followed by the bytes. Ingot computes the same, so this is also the id of the file in any repository that holds it. Try echo -n hello | git hash-object --stdin beside it.

CL-USER> (ingot:hash-object :blob
                            (ingot:string-to-octets "hello"))
"b6fc4c620b67d95f953a5c1c1230aaab5db5a1b0"

Git commands and their Ingot forms

Ingot is the engine, not the porcelain, but most of what one types at git has a form here. *repo* is an open repository, ids are forty-character strings, and a commit is read with read-commit and asked with the commit- readers.

git	Ingot
git init --bare notes.git	(init-repository "notes.git/")
git rev-parse HEAD, git rev-parse topic	(head-oid *repo*), (resolve *repo* "topic")
git branch, git tag	(branches *repo*), (tags *repo*)
git log -10 -- src/	(log-commits *repo* oid :limit 10 :path "src/")
git show <commit>	(read-commit *repo* oid), (commit-diff *repo* oid)
git ls-tree -r <tree>	(tree-paths *repo* tree-oid)
git show <commit>:path/file	(blob-at *repo* commit-oid "path/file")
git hash-object file	(hash-object :blob octets)
git add and git commit	(make-commit! *repo* :branch "master" :author who :message "..." :actions '((:action :create :path "f" :content "...")))
git checkout -b topic master, then a commit	(make-commit! *repo* :branch "topic" :start-branch "master" ...)
git diff a b -- file	(unified-diff old-text new-text); whole trees: (tree-diff *repo* old-tree new-tree)
git merge-base a b	(merge-base *repo* a b)
git merge topic	(can-merge? *repo* src dst), then (merge-branches! *repo* "topic" "master" :author who)
git update-ref, git branch -D	(update-ref! *repo* "refs/heads/x" oid :old old), (delete-ref! *repo* "refs/heads/x")
git symbolic-ref HEAD refs/heads/main	(set-default-branch! *repo* "main")
git archive --format=tar.gz --prefix=p/ <tree>	(tar-gz-tree *repo* tree-oid :prefix "p/"), (zip-tree ...)
git fetch, git clone	(fetch! *repo* transport), with a transport your HTTP client lends
serving git-upload-pack and git-receive-pack	(advertise-refs *repo* :upload-pack), (upload-pack *repo* body), (receive-pack *repo* body :check ...)

Reading a repository you already have

open-repository takes a checkout or a bare repository and reads it as git left it, packs and all:

(defparameter *repo* (ingot:open-repository "/path/to/a/checkout/"))
(ingot:default-branch *repo*)
(ingot:branches *repo*)
(mapcar #'ingot:commit-subject
        (ingot:log-commits *repo* (ingot:head-oid *repo*) :limit 10))
(ingot:tags *repo*)

A path filter on the log, :path "src/", answers the commits that touched it, as git log -- src/ does.

Under the hood

An object is found loose first, then in every pack of the repository through its index (the fanout and the sorted ids), a delta resolved against its base by offset or by id. A ref update is written through a temporary file and a rename, compare-and-swap against the old value under the repository's lock, so two writers do not lose each other. A commit is made from a base tree and a list of actions, which is what a web editor or an API would do; there is no work tree and no index. A fetch answered by Ingot copies the entries of its own packs as they lie, deltas included when the base goes too, so a clone of a pushed repository gets a pack much the size of the one that was pushed.

Every function signals ingot:git-error (or not-found or ref-conflict) rather than answering half a thing.

What is not there

git status, git add and the index: Ingot keeps no work tree. SHA-256 repositories (the id is twenty bytes throughout). Shallow clones. Deltas of its own making: a pack it writes reuses the deltas of the packs it holds and writes the rest whole, and nothing repacks. Rename detection in diffs. The contents of submodules. Each is a seam rather than a wall.