Introduction

Rory Graves @a_dev_musing

Rory was writing mobile games when Nokia Snake was the height of technology, long before it was cool.

He was doing novel research on dynamic network protocols to replace TCP/IP, which was when a cluster was more likely to refer to a collection of peanuts than a collection of server computers.

He shows people around an old windmill in southern England at the weekends.

And he is a black belt in a martial art that I cannot pronounce, with the superpower of sword fighting.

Compiling

So believe me, you really don’t want to be waiting on your compiler with Rory around!

Sam Halliday @fommil

Mun

FreeHighSchoolScienceTexts No Evidence he ever got back from the Mun

Straw Poll

TL;DR

  1. it’s not another IDE, it’s about context
  2. so people can use their favourite text editor
  3. so it can be used programmatically for better tooling / analysis

Overview

End with ‘Demo Time!’

Demo


(set-frame-font “Inconsolata-24”) (required ’command-log-mode t) (global-command-log-mode) (setq guide-key/guide-key-sequence nil)

sleep 2 ; recordmydesktop –no-sound –on-the-fly-encoding –no-frame -o ensime.ogv

Rough script, for a walkthrough:

  1. start with scala-mode and show regex syntax highlighting.
  2. turn on ensime and comment on the sbt deploy process.
  3. wait for the funny and announce that we’ll have a call for new funnies for the 1.0 release, with a prize for the best.
  4. semantic highlighting kicks in and we can see that everything is correctly detected where regex failed. Until I started using ENSIME for Scala, I always thought the colour coding was a bit random, but now I use colour to help me understand code and I can get a feel for it based on whether its type heavy, data driven or lots of functions.
  5. introduce and error by creating an instance of something that hasn’t been imported. Show the error and then import a suggestion.
  6. Then show the type, and inspect the type.
  7. Come back to the code buffer and autocomplete a method and show the parameter autofill / tab jumping.
  8. Generate / go to the corresponding test
  9. Call out to sbt to run just that test file.
  10. Back to the test file, open up a nearby test file and use it to jump to the implementation.
  11. Pick a class in a third party library and jump to its source code.
  12. Come back, ask for its documentation, view it in your browser. Isn’t it a pity that nobody writes Scaladocs!
  13. Ask for the documentation for something in the Java standard library, read it in the browser.
  14. Rename a local variable or method with refactor.
  15. Reformat the code with Scalariform.
  16. Start a REPL, type some stuff.
  17. Search for a class, show that it works with camel case.

Architecture


This is an architectural overview of the internals of the ensime-server, which is bounded here by the dotted lines.

The text editor communicates with the server via SWANK, which is a bidirectional TCP/IP sockets protocol using S-Expressions as the language. This is really convenient for emacs. We’ll talk about protocols later because we really don’t see ENSIME as being an emacs only library.

The server runs locally, so it also has direct access to the files on the disc and can watch for changes without needing to be told about them. This is typically used for detecting changes in the compiled files rather than looking for changes in source code.

And when the server is started, it needs to be given a .ensime file which defines the project layout. This is typically generated by the build tool.

Inside the server, everything goes via the central Project class which effectively just delegates to the relevant sub-component. The two big parts are the Search Service and the Analyzer:

  1. The Search Service indexes all the binaries related to the project, including third party jars. We use ASM to do the heavy lifting and we persist the results to H2 to enable various types of searches. We also build up an index in Lucene for advanced searching, such as camel case searching of a classname.
  2. The Analyzer is our layer that sits on top of the Scala Presentation Compiler, which is an interactive version of the Scala Compiler but is supposed to be quicker because it shortcuts various stages in order to be responsive. This is the same backend that is used by the Scala IDE, but it is released as part of the official Scala Compiler jar.
  3. We also have the ability to identify source code to binaries, e.g. to relate your third party source zip files to the jars that you’re including. This lets us implement the “jump to source” functionality beyond the user’s project files.
  4. Documentation is hosted via a Spray HTTP server and viewed in a normal web browser.
  5. A debug manager component allows interactive debugging sessions against a running JVM. It manages the state of the threads and allows stepping and inspection.
  6. Then there is some stuff that lives on the cusp of what ensime should really do, like source formatting. You can issue a command to ENSIME and it’ll format your sources, which works well if you don’t have automatic formatting as part of your build.

We’ve had various discussions about where the “in/out” line should be and we decided that compilation is definitely on the other side of the line because that’s what build tools are designed to do. But with the possibility of closer integration with the new sbt server, we will see the coupling with sbt, in particular, becoming stronger. But we’re not limiting ourselves to one build tool because the reality is that a large number of corporate development environments are using legacy build systems and we want to support that.

Ecosystem

Projects

ENSIME isn’t just one project. The server is a Scala project with separate branches to support the three active stable release versions of Scala. (Yes, people are still using Scala 2.9 in production systems).

The compiler API has been quite volatile across versions, so we are not able to offer a single source build. The server jars are published as snapshots to Sonatype, more about that in a few slides.

The emacs client is a completely separate project to the server. It has its own github repository and is published using the emacs package management system MELPA.

Build tool plugins also live in their own repositories. The functionality offered here is typically the ability to generate the .ensime file, although we might see more functionality in the future, for example, integration with the sbt server or automating the debugging launch process with build tool commands. The plugins are rarely updated, so they are published to Maven Central.

And we have a separate repository for creating our docker images, which is used by the build system. We have an official docker image that is used for each release branch and that has all our hard dependencies pre-loaded (i.e. JDK and sbt), this is published to Docker Hub.

Community

c.f. Benjamin Mako Hill’s talk at LibrePlanet 2013

Benjamin Mako Hill gave a talk at LibrePlanet 2013 did an analysis of projects on github. He showed that the vast majority of projects have one contributor.

ENSIME is a really nice project to work on because it doesn’t have a single owner. Aemon started the project way back in 2010, but many people have been helping out over the last few years.

We don’t have a financial backer, so realistic the only way for ENSIME to survive and grow is to have a strong community.

Community Principles

Fundamentally, a good community has to be inclusive, share a common goal, and be fun. As admins we try to encourage that vibe with anyone who comes across us and we have a formal code of conduct, as we are members of Typelevel.

Typically when we get a bug report, or a feature request, we’ll offer to help the reporter. If you raise a ticket, you can expect to get pointed at the general area in the code that needs improvement and any changes you make will be quickly reviewed. Often with an avalanche of additional suggestions! :-)

This approach obviously doesn’t always work out, and some tickets might never get closed, but clearly it’s working out pretty well because we get a *lot* of contributors sending one or two PRs and make an improvement.

We take a pragmatic approach to feature and bugfix prioritisation. It’s not like we’re developing a mass consumer website, or a specialist system with business users — everybody who uses ENSIME is a scala developer and should know how to interpret an exception in the logs. That’s why we prefer simple solutions and clear error messages to handling corner case failures that can be fixed by the user.

We try to maintain a presence at the Hack the Tower hackday events in London. Free burgers for anybody who comes along and hacks on ENSIME, and please join us remotely and we’ll find you something to do.

Continuous Integration

Key to survival of ENSIME is ease of contribution, and the only way to manage that is by automating as much of the development process as possible.

We have a strong emphasis on testing (unit, integration, client tests), coverage, shippable docker images (soon to be replaced by drone).

Continuous Delivery

We can do this because of the suite of tests.

Green master builds are automatically deployed to sonatype snapshots repository, and we use sbt in the emacs client to do the deployment on the client machine and obtain the classpath. Every time the client updates, or the user types ensime-update the server deployment will be refreshed.

Website

We now have a new website which will become the focus for all documentation over the coming months.

A lot of pages have TODO notes on them at the moment, and that’s not a typo. Every page has an edit button on it and in the community spirit, we hope that the TODOs will be replaced by real content by users and developers of ENSIME shortly.

What going on

This year

The Future

1.0

Beyond thunderdome!

Javapocolpyse

2.0

Graphpocolpyse

Graphpocolpyse 2

Presentation Compiler Interface Rewrite

Sharing for the win!

Conclusion

Conclusion

Questions?

www.ensime.org

Come to the hacking session this afternoon!