ENSIME

Rory Graves and Sam Halliday

Video from Scala Days 2015

Introduction

Rory Graves @a_dev_musing

  • Wrote mobile games before it was cool
  • Dynamic networks before clusters were cool
  • Shows people around an old windmill at weekends
  • Martial artist who is a trained swordsman

Sam Halliday @fommil

  • co-founded FHSST, 5mil Free textbooks in South Africa
  • co-founded a mathematics company
    • quantum mechanics, machine learning, etc
  • co-founded Neurofiction: brain scanners + fiction
  • netlib-java underpinning Apache Spark
  • most proud of getting to the Mun in Kerbal…

Straw Poll

  • IntelliJ
  • ScalaIDE
  • Emacs
  • Other

TL;DR

  • Ensime is not really an IDE
  • Its a toolset for building IDEs and analysing code.
  • But…

It is time to build our own IDE!

Overview

  • Ensime in action
  • Architecture
  • Ecosystem - our community
  • Protocols - Shapeless marshalling magic
  • The Roadmap
  • The Future

Demo

Architecture

Ecosystem

Projects

Community

c.f. Benjamin Mako Hill’s talk at LibrePlanet 2013

Server Contributors

Author + lines - lines
Aemon Cannon 62529 47989
Sam Halliday 17505 17565
Rory Graves 12080 10156
Eric Daniel 2958 1113
Eugene Burmako 413 151
Jason Fager 230 144
Marc Saegesser 160 324
Fabian 144 45
Alexander Pupeikis 126 20
Jacob Schlather 66 64

Matt Russell, John Sullivan, Daniel Spiewak, Radzisław Galler, Toshiyuki Takahashi, Alexander Baier, Peluko, Grégoire Neuville, Igor Shymko, Anatoly Fayngelerin, tbje, Tomás Senart, Mark Schaake, Marc Weber, Felix Geller, toshiyuki takahashi, sksamuel, pashky, William O’Hanley, Wilfred Springer, The Gitter Badger, Steve Jenson, Scalariform, RayRacine, Pawel Kopiczko, Mike O’Connor, Michael Terry, Jack Viers, Ivan Poliakov, Huw Giddens, Hubert Plociniczak, Habibullah Pagarkar, Evgeny Chukreev, Erik Osheim, Eric Sessoms, Deokhwan Kim, Dave Fayram, Bozhidar Batsov, Bas Kok, Alexandre Bertails, Alejandro Pedraza.

Emacs Contributors

Author + lines - lines
Aemon Cannon 30959 15624
Eric Daniel 16095 15625
Sam Halliday 2098 4131
Radzisław Galler 1094 3183
Grégoire Neuville 302 143
Alexander Pupeikis 132 19
Toshiyuki Takahashi 69 62
Fabian 59 9
Alexander Baier 57 58

Robin Green, Łukasz Klich, Paul Sexton, Kirill Kulikov, Erik Assum, Marc Saegesser, ScottyB, Peluko, Eugene Burmako, Matt Russell, Daniel Spiewak, John Sullivan, Jason Fager, Igor Shymko, Anatoly Fayngelerin, tbje, Tyson Hamilton, Tomás Senart, MrBones118, Mark Schaake, Marc Weber, Marc A. Saegesser, Felix Geller, Alejandro Pedraza, toshiyuki takahashi, pdn, pashky, jules, hmgibson23, Wilfred Springer, Sviridov Alexander, Steve Jenson, Scalariform, Sacha Chua, Rory Graves, RayRacine, Mike O’Connor, Michael Terry, Jack Viers, Ivan Poliakov, Huw Giddens, Hubert Plociniczak, Howard Branch, Habibullah Pagarkar, Greg Pfeil, Evgeny Chukreev, Erik Osheim, Deokhwan Kim, Dave Fayram, Dave Aitken, Bozhidar Batsov, Ben Spencer, Bas Kok, Andre Silva, Alexandre Bertails.

Community Principles

  • Code of Conduct
    • good vibes
  • Lots of help for new contributors
    • ticket → general area of code
    • fast PR reviews and feedback
  • Pragmatic approach
    • everybody is a Scala dev
    • KISS: prefer failure / explanation, not complexity
  • Bounties!
    • failed experiment

Continuous Integration

Continuous Delivery

Protocol

Greenspun’s tenth rule

“Any sufficiently complicated program contains an ad hoc, informally-specified, bug-ridden, slow implementation of half of Common Lisp.”

“The class of sufficiently complicated programs includes Common Lisp.”

— Robert Morris’ corollary (Y-Combinator)

S-Expressions

(a . (b . (c . nil)))

(a b c) ;; list syntax
(:keyA . (valueA . (:keyB . (valueB . nil))))

(:keyA valueA
 :keyB valueB) ;; data syntax with keywords

(:file "Foo.scala"
 :line 13)

;; complex map structure
((1 2 3) "Foo.scala"
 (:key value) 13)

SWANK

  • ENSIME derived from SWANK — easy for Emacs
  • vs JSON
    • JSON keys are String
    • JSON maps are unordered
    • S-Exp naturally encodes structure
    • neither has an official schema
case class TypeAtPointReq(
  file: File,
  range: OffsetRange
) extends RpcTypeRequest
case class BasicTypeInfo(
  name: String,
  typeId: Int,
  declAs: DeclaredAs,
  fullName: String,
  typeArgs: Iterable[TypeInfo],
  members: Iterable[EntityInfo],
  pos: Option[SourcePosition],
  outerTypeId: Option[Int]
) extends TypeInfo
(:swank-rpc
 (swank:type-at-point
  "<...>/org/ensime/indexer/SearchService.scala"
  1858)
 7)
(:return
 (:ok
   (:arrow-type nil
    :name "String"
    :type-id 7
    :decl-as class
    :full-name "java.lang.String"
    :type-args nil
    :members nil
    :pos (:type line
          :file "<...>/java/lang/String.java"
          :line 134)
    :outer-type-id nil))
 7)

S-Express

  • inspired by spray-json
    • but with shapeless magic for Product
  • helps automate addition of endpoints
    • makes it easier to contribute features!
    • but backwards compatibility is limiting

JSON

Why JSON?

  • Most editors don’t know how to speak Lisp!
  • If you build it, they will come.
  • JERK

Shapeless

  • HListcase class
  • Coproductsealed trait
  implicit def familyFormat[T](
    implicit
    gen: LabelledGeneric[T],
    sg: WrappedRootJsonFormat[T, gen.Repr],
    tpe: Typeable[T]
  ): RootJsonFormat[T] = new RootJsonFormat[T] {
    if (log.isTraceEnabled)
      log.trace(s"creating ${tpe.describe}")

    def read(j: JsValue): T = gen.from(sg.value.read(j))
    def write(t: T): JsObject = sg.value.write(gen.to(t))
  }

Hipster Aux

  implicit def familyFormat[T, Repr](
    implicit
    gen: LabelledGeneric.Aux[T, Repr],
    sg: Lazy[WrappedRootJsonFormat[T, Repr]],
    tpe: Typeable[T]
  ): RootJsonFormat[T] = new RootJsonFormat[T] {
    if (log.isTraceEnabled)
      log.trace(s"creating ${tpe.describe}")

    def read(j: JsValue): T = gen.from(sg.value.read(j))
    def write(t: T): JsObject = sg.value.write(gen.to(t))
  }
  sealed trait SimpleTrait
  case class Foo(s: String) extends SimpleTrait
  case class Bar() extends SimpleTrait
  case object Baz extends SimpleTrait

  Foo("foo").toJson // {"type":"Foo","s":"foo"}
  Bar().toJson      // {"type":"Bar"}
  Baz.toJson        // {"type":"Baz"}

Compile Times

Roadmap

Performance

  • Ensime is a shim over the presentation compiler
  • Profiling shows compiler performance is poor in places.
  • e.g. scala.reflect.internal.Constants.safeToString generates 100s of temporary objects to create a single result string.
  • Performance is my next primary target

1.0 Stable

2.0 Simplifications / Java

Whats next?

  • Simplification
    • Core
    • Protocol
  • design for enhancement
  • Going reactive
  • Reverse lookup / Find Implementations
  • Java Support - Enjime

The future

More editors

  • Sublime
  • Atom
  • vi The editor of the beast
  • Your editor here

Atom Demo

Viktor Hedefalk

Incoming fire!

  • TASTY
  • Scala.Meta
  • sbt-remote-control

ENSIME as a toolkit

  • A user level API over the compiler
    • Analysis
    • Tooling
      • Dead code finding and cleanup
      • Refactoring / Hints

Ensime-IDE

  • A fundamentally different model
    • IDEs tend to be monolithic
    • and tied to a legacy model (Java)

How?

Components:

  • Web based front end - Atom/ScalaJS
  • Shared marshalling code (JERK/Shapeless)
  • Ensime based Server (Docker deployed)
  • Nodes to to tasked (sbt-remote/ Docker)
    • Compile
    • Repl
    • Testing
    • Debugging
    • Coverage

Shiny new things

  • Project separation
  • Remotely hosted development
    • Remote working
  • Scalable
    • spin up an entire farm

Collaboration

  • Instant setup
    • Setup time - insta-project
    • training setup time
    • onboarding
  • Instant Collaboration
    • Pairing

Other possibilities

  • Scala Playground
    • 4scala
  • Crowdsourcing

Final thoughts

Final thoughts

  • Ensime:
    • is a community project
    • opens up a bunch of interesting possibilities.

Questions?

Come join the party!

https://github.com/ensime/ensime-server