Language Server Protocol Tutorial: From VSCode to Vim
The main artifact of all your work is most likely plain text files. So why don’t you use Notepad to create them? Find out how the Language Server Protocol can transform text editors into code editors, without coupling.
The main artifact of all your work is most likely plain text files. So why don’t you use Notepad to create them? Find out how the Language Server Protocol can transform text editors into code editors, without coupling.
An active open source contributor, Jeremy specializes in advanced JavaScript. He believes in clean code, testing, and reading the manual.
PREVIOUSLY AT
The main artifact of all your work is most likely plain text files. So why don’t you use Notepad to create them?
Syntax highlighting and automatic formatting are just the tip of the iceberg. What about linting, code completion, and semi-automatic refactoring? These are all very good reasons to use a “real” code editor. These are vital to our day-to-day, but do we understand how they work?
In this Language Server Protocol tutorial, we’ll explore these questions a bit and find out what makes our text editors tick. In the end, together we’ll implement a basic language server along with example clients for VSCode, Sublime Text 3, and Vim.
Compilers vs. Language Services
We’ll skip over syntax highlighting and formatting for now, which is handled with static analysis—an interesting topic in its own right—and focus on the main feedback we get from these tools. There are two main categories: compilers and language services.
Compilers take in your source code and spit out a different form. If the code does not follow the language’s rules, the compiler will return errors. These are quite familiar. The problem with this is it’s usually quite slow and limited in scope. How about offering assistance while you’re still creating the code?
This is what language services provide. They can give you insights into your codebase while it’s still in the works, and probably a lot faster than compiling the whole project.
The scope of these services is varied. It can be something as simple as returning a list of all the symbols in the project, or something complex like returning steps to refactor code. These services are the primary reason we use our code editors. If we just wanted to compile and see errors, we could do that with a few keystrokes. Language services give us more insights, and very quickly.
Betting on a Text Editor for Programming
Notice that we have not called out specific text editors yet. Let’s explain why with an example.
Say you’ve developed a new programming language called Lapine. It’s a beautiful language and the compiler gives terrific Elm-like error messages. Additionally, you are able to provide code completion, references, refactoring help, and diagnostics.
Which code/text editor do you support first? What about after that? You’ve got an uphill battle fighting to get people to adopt it, so you want to make it as easy as possible. You don’t want to pick the wrong editor and miss out on users. What if you keep your distance from the code editors and focus on your specialty—the language and its features?
Language Servers
Enter language servers. These are tools that talk to language clients and provide the insights we’ve mentioned. They are independent of text editors for the reasons we just described with our hypothetical situation.
As usual, another layer of abstraction is just what we need. These promise to break the tight coupling of language tools and code editors. Language creators can wrap their features in a server once, and code/text editors can add small extensions to turn themselves into clients. It’s a win for everyone. To facilitate this, though, we need to agree on how these clients and servers will communicate.
Lucky for us, this isn’t hypothetical. Microsoft has already begun by defining the Language Server Protocol.
As with most great ideas, it grew out of necessity rather than foresight. Many code editors had already started adding support for various language features; some features outsourced to third-party tools, some done under the hood within the editors. Scalability issues came about, and Microsoft took the lead on splitting things. Yes, Microsoft paved the way to move these features out of the code editors rather than hoarding them within VSCode. They could have kept building their editor, locking in users—but they set them free.
Language Server Protocol
The Language Server Protocol (LSP) was defined in 2016 to help separate language tools and editors. There are still many VSCode fingerprints on it, but it is a major step in the direction of editor agnosticism. Let’s examine the protocol a bit.
Clients and servers—think code editors and language tools—communicate in simple text messages. These messages have HTTP-like headers, JSON-RPC content, and may originate from either the client or server. The JSON-RPC protocol defines requests, responses, and notifications and a few basic rules around them. A key feature is that it is designed to work asynchronously, so clients/servers can deal with messages out of order and with a degree of parallelism.
In short, JSON-RPC allows a client to request another program to run a method with parameters and return a result or an error. LSP builds on this and defines the methods available, expected data structures, and a few more rules around the transactions. For example, there’s a handshake process when the client starts up the server.
The server is stateful and only meant to handle a single client at a time. There are no explicit restrictions on communication, though, so a language server could run on a different machine than the client. In practice, that would be pretty slow for real-time feedback, though. Language servers and clients work with the same files and are pretty chatty.
The LSP has a decent amount of documentation once you know what to look for. As mentioned, much of this is written within the context of VSCode, though the ideas have a much broader application. For example, the protocol specification is all written in TypeScript. To aid explorers unfamiliar with VSCode and TypeScript, here’s a primer.
LSP Message Types
There are many groups of messages defined in the Language Server Protocol. They can be roughly divided into “admin” and “language features.” Admin messages contain those used in the client/server handshake, opening/altering files, etc. Importantly, this is where clients and servers share which features they handle. Certainly, different languages and tools offer different features. This also allows for incremental adoption. Langserver.org names a half-dozen key features that clients and servers should support, at least one of which is required to make the list.
Language features are what we’re mostly interested in. Of these, there is one to call out specifically: the diagnostic message. Diagnostics are one of the key features. When you open a file, it’s mostly assumed that this will run. Your editor should tell you if there is something wrong with the file. The way this happens with LSP is:
- The client opens the file and sends
textDocument/didOpen
to the server. - The server analyzes the file and sends the
textDocument/publishDiagnostics
notification. - The client parses the results and displays error indicators in the editor.
This is a passive way to get insights from your language services. A more active example would be finding all the references for the symbol under your cursor. This would go something like:
- The client sends
textDocument/references
to the server, specifying a location in a file. - The server figures out the symbol, locates references in this and other files, and responds with a list.
- The client displays the references to the user.
A Blacklist Tool
We could surely dig into the specifics of the Language Server Protocol, but let’s leave that for client implementers. To cement the idea of editor and language tool separation, we’ll play the role of tool creator.
We will keep it simple and, instead of creating a new language and features, we will stick to diagnostics. Diagnostics are a good fit: They’re just warnings about a file’s content. A linter returns diagnostics. We’ll make something similar.
We will make a tool to notify us of words we would like to avoid. Then, we’ll provide that functionality to a couple of different text editors.
The Language Server
First, the tool. We’ll bake this right into a language server. For simplicity, this will be a Node.js app, though we could do it with any tech able to use streams for reading and writing.
Here is the logic. Given some text, this method returns an array of the matched blacklisted words and the indices where they were found.
const getBlacklisted = (text) => {
const blacklist = [
'foo',
'bar',
'baz',
]
const regex = new RegExp(`\\b(${blacklist.join('|')})\\b`, 'gi')
const results = []
while ((matches = regex.exec(text)) && results.length < 100) {
results.push({
value: matches[0],
index: matches.index,
})
}
return results
}
Now, let’s make it a server.
const {
TextDocuments,
createConnection,
} = require('vscode-languageserver')
const {TextDocument} = require('vscode-languageserver-textdocument')
const connection = createConnection()
const documents = new TextDocuments(TextDocument)
connection.onInitialize(() => ({
capabilities: {
textDocumentSync: documents.syncKind,
},
}))
documents.listen(connection)
connection.listen()
Here, we are utilizing the vscode-languageserver
. The name is misleading, as it can certainly work outside VSCode. This is one of the many “fingerprints” you see of LSP’s origins. vscode-languageserver
takes care of the lower-level protocol and allows you to focus on the use cases. This snippet starts a connection and ties it into a document manager. When a client connects to the server, the server will tell it that it would like to be notified of text documents being opened.
We could stop here. This is a fully functioning, albeit pointless, LSP server. Instead, let’s respond to document changes with some diagnostic information.
documents.onDidChangeContent(change => {
connection.sendDiagnostics({
uri: change.document.uri,
diagnostics: getDiagnostics(change.document),
})
})
Finally, we connect the dots between the document that changed, our logic, and the diagnostics response.
const getDiagnostics = (textDocument) =>
getBlacklisted(textDocument.getText())
.map(blacklistToDiagnostic(textDocument))
const {
DiagnosticSeverity,
} = require('vscode-languageserver')
const blacklistToDiagnostic = (textDocument) => ({ index, value }) => ({
severity: DiagnosticSeverity.Warning,
range: {
start: textDocument.positionAt(index),
end: textDocument.positionAt(index + value.length),
},
message: `${value} is blacklisted.`,
source: 'Blacklister',
})
Our diagnostics payload will be the result of running the document’s text through our function, then mapped to the format expected by the client.
This script will create all that for you.
curl -o- https://raw.githubusercontent.com/reergymerej/lsp-article-resources/revision-for-6.0.0/blacklist-server-install.sh | bash
Note: If you’re uncomfortable with strangers adding executables to your machine, please check the source. It creates the project, downloads index.js
, and npm link
s it for you.
Complete Server Source
The final blacklist-server
source is:
#!/usr/bin/env node
const {
DiagnosticSeverity,
TextDocuments,
createConnection,
} = require('vscode-languageserver')
const {TextDocument} = require('vscode-languageserver-textdocument')
const getBlacklisted = (text) => {
const blacklist = [
'foo',
'bar',
'baz',
]
const regex = new RegExp(`\\b(${blacklist.join('|')})\\b`, 'gi')
const results = []
while ((matches = regex.exec(text)) && results.length < 100) {
results.push({
value: matches[0],
index: matches.index,
})
}
return results
}
const blacklistToDiagnostic = (textDocument) => ({ index, value }) => ({
severity: DiagnosticSeverity.Warning,
range: {
start: textDocument.positionAt(index),
end: textDocument.positionAt(index + value.length),
},
message: `${value} is blacklisted.`,
source: 'Blacklister',
})
const getDiagnostics = (textDocument) =>
getBlacklisted(textDocument.getText())
.map(blacklistToDiagnostic(textDocument))
const connection = createConnection()
const documents = new TextDocuments(TextDocument)
connection.onInitialize(() => ({
capabilities: {
textDocumentSync: documents.syncKind,
},
}))
documents.onDidChangeContent(change => {
connection.sendDiagnostics({
uri: change.document.uri,
diagnostics: getDiagnostics(change.document),
})
})
documents.listen(connection)
connection.listen()
Language Server Protocol Tutorial: Time for a Test Drive
After the project is link
ed, try running the server, specifying stdio
as the transport mechanism:
blacklist-server --stdio
It’s listening on stdio
now for the LSP messages we talked about before. We could provide those manually, but let’s create a client instead.
Language Client: VSCode
As this technology originated in VSCode, it seems appropriate to start there. We’ll create an extension that will create an LSP client and connect it to the server we just made.
There are a number of ways to create a VSCode extension, including using Yeoman and the appropriate generator, generator-code
. For simplicity, though, let’s do a barebones example.
Let’s clone the boilerplate and install its dependencies:
git clone git@github.com:reergymerej/standalone-vscode-ext.git blacklist-vscode
cd blacklist-vscode
npm i # or yarn
Open the blacklist-vscode
directory in VSCode.
Press F5 to start another VSCode instance, debugging the extension.
In the first VSCode instance’s “debug console,” you will see the text, “Look, ma. An extension!”
We’ve now got a basic VSCode extension working without all the bells and whistles. Let’s make it an LSP client. Close both VSCode instances and from within the blacklist-vscode
directory, run:
npm i vscode-languageclient
Replace extension.js with:
const { LanguageClient } = require('vscode-languageclient')
module.exports = {
activate(context) {
const executable = {
command: 'blacklist-server',
args: ['--stdio'],
}
const serverOptions = {
run: executable,
debug: executable,
}
const clientOptions = {
documentSelector: [{
scheme: 'file',
language: 'plaintext',
}],
}
const client = new LanguageClient(
'blacklist-extension-id',
'Blacklister',
serverOptions,
clientOptions
)
context.subscriptions.push(client.start())
},
}
This uses the vscode-languageclient
package to create an LSP client within VSCode. Unlike vscode-languageserver
, this is tightly coupled to VSCode. In short, what we’re doing in this extension is creating a client and telling it to use the server we created in the previous steps. Glossing over the VSCode extension specifics, we can see that we’re telling it to use this LSP client for plain text files.
To test drive it, open the blacklist-vscode
directory in VSCode. Press F5 to start another instance, debugging the extension.
In the new VSCode instance, create a plain text file and save it. Type “foo” or “bar” and wait a moment. You will see warnings that these are blacklisted.
That’s it! We didn’t have to recreate any of our logic, just coordinate the client and server.
Let’s do it again for another editor, this time Sublime Text 3. The process will be quite similar and a little easier.
Language Client: Sublime Text 3
First, open ST3 and open the command palette. We need a framework to make the editor an LSP client. Type “Package Control: Install Package” and hit enter. Find the package “LSP” and install it. Once complete, we have the ability to specify LSP clients. There are many presets, but we’re not going to use those. We’ve created our own.
Again, open the command palette. Find “Preferences: LSP Settings” and hit enter. This will open the configuration file, LSP.sublime-settings
, for the LSP package. To add a custom client, use the configuration below.
{
"clients": {
"blacklister": {
"command": [
"blacklist-server",
"--stdio"
],
"enabled": true,
"languages": [
{
"syntaxes": [
"Plain text"
]
}
]
}
},
"log_debug": true
}
This may look familiar from the VSCode extension. We defined a client, told it to work on plain text files, and specified the language server.
Save the settings, then create and save a plain text file. Type “foo” or “bar” and wait. Again, you’ll see warnings that these are blacklisted. The treatment—how the messages are displayed in the editor—is different. However, our functionality is the same. We barely even did anything this time to add support to the editor.
Language “Client”: Vim
If you’re still not convinced that this separation of concerns makes it easy to share features across text editors, here are the steps to add the same functionality to Vim via Coc.
Open Vim and type :CocConfig
, then add:
"languageserver": {
"blacklister": {
"command": "blacklist-server",
"args": ["--stdio"],
"filetypes": ["text"]
}
}
Done.
Client-server Separation Lets Languages and Language Services Thrive
Separating the responsibility of language services from the text editors they are used in is clearly a win. It allows language feature creators to focus on their specialty and editor creators to do the same. It’s a fairly new idea, but adoption is spreading.
Now that you’ve got a basis to work from, maybe you can find a project and help move this idea forward. The editor flame war will never end, but that’s OK. As long as the language abilities can exist outside specific editors, you are free to use whatever editor you like.
As a Microsoft Gold Partner, Toptal is your elite network of Microsoft experts. Build high-performing teams with the experts you need - anywhere and exactly when you need them!
Further Reading on the Toptal Blog:
Understanding the basics
What is a language server protocol?
The Language Server Protocol is a set of rules describing how language clients and language servers should communicate. It was defined in 2016 to help decouple editors from language-specific tools like tsserver.
Which languages are compiled?
C++, Java, Erlang, Go, and Common Lisp are all compiled languages, which means their source is converted into another form, like machine code, prior to execution. Interpreted languages, like Python or JavaScript, have source code run through an interpreter as their programs are executed.
What is the role of the compiler?
A compiler’s role is to convert source code into another form for execution, typically machine code. This differs from a transpiler, which converts source code into another type of source code, like TypeScript to JavaScript. A compiler will report errors in source code.
What are the main functions of a compiler?
A compiler’s main function is to convert source code to machine code. This includes lexing and parsing to build a model of the program, implementing optimizations, and generating the target code. The front-end analysis provides feedback that can help developers identify errors in syntax.
What are text editors used for?
Text editors are programs used to modify text files, e.g., Notepad, Word, Notepad++, Sublime Text, Vim, and Emacs. Typically, developers avoid using simple text editors for programming unless they can be extended to understand text within the context of a programming language, effectively turning them into IDEs.
What is VSCode made with?
VSCode is made with Node.js, via Electron. The source is available on Github under Microsoft’s repository listing.
Jeremy Greer
Orlando, FL, United States
Member since April 12, 2016
About the author
An active open source contributor, Jeremy specializes in advanced JavaScript. He believes in clean code, testing, and reading the manual.
PREVIOUSLY AT