Barak Pearlmutter: Axis of eval!

Axis of eval!

Automatic differentiation mates with lambda calculus birthing monster compiler faster than fortran.

Barak Pearlmutter

Abstract: The technique known in the machine learning community as "backpropagation" is a special case of "reverse-mode accumulation automatic differentiation", or "reverse AD". We will explore forward and reverse AD using a novel formulation based on differential geometry. In this formulation, the AD operators naturally generalize to a much broader range of computer programs, including programs containing iterate-to-fixedpoint loops; invoking or embodying higher-order functions; invoking optimizers; or even themselves invoking AD operators. Algorithms including fast exact Hessian-vector multiplication, Pineda/Almeida fixedpoint backpropagation, and a wide variety of other techniques can be defined and implemented as one-liners using these generalized AD operators. These methods allow very complicated systems, like bi-level optimization architectures, to be built and optimized using gradient methods. This system has been formalized using the tools of modern Programming Language Theory, and a research prototype implementation has been constructed which exhibits startlingly good (i.e., FORTRAN-like) numeric performance.

(Joint work with Jeffrey Mark Siskind)

Vabljeni!