Using Scala.js With NPM And Browserify

If you use Scala.js, the compiler of the Scala language to JavaScript, you may find the standard dependency management of Scala.js too limiting in the modern JavaScript world. Scala.js manages dependencies with WebJars, while JavaScript developers manage dependencies using NPM. Since dependencies produced by NPM are server-side, usually an additional step using Browserify or Webpack to generate browser code is needed.

In this post, I will describe how to integrate Scala.js with the plethora of JavaScript modules available on NPM. You can check this GitHub repository for a working example of the techniques described here. Using this example and by reading this post, you will be able to collect your JavaScript libraries using NPM, create a bundle using Browserify, and use the result in your own Scala.js project. All this without even installing Node.js, as everything is managed by SBT.

Managing Dependencies for Scala.js

Today, writing applications in languages which compile to JavaScript is becoming a very common practice. More and more people are moving to extended JavaScript languages, like CoffeeScript or TypeScript, or transpilers like Babel, which allow you to use ES6 today. At the same time, Google Web Toolkit (a compiler from Java to JavaScript) is mostly used for enterprise applications. For these reasons, as a Scala developer I do not consider anymore using Scala.js to be a strange choice. The compiler is fast, the produced code is efficient, and overall it is just a way to use the same language both on the front-end and back-end.

That said, using Scala tools in the JavaScript world is not yet 100 percent natural. You sometimes have to fill the gap from the JavaScript ecosystem to the Scala one. Scala.js as a language has an excellent interoperability with JavaScript. Because Scala.js is a compiler for the Scala language to JavaScript, it is very easy to interface Scala code to existing JavaScript code. Most importantly, Scala.js gives you the ability to create typed interfaces (or facades) to access untyped JavaScript libraries (similar to what you do with TypeScript). For developers used to strongly typed languages like Java, Scala or even Haskell, JavaScript is too loosely typed. If you are such a developer, probably the main reason is because you may want to use Scala.js is to get a (strongly) typed language on top of an untyped language.

A problem in the standard Scala.js toolchain, which is based on SBT and is still left somewhat open, is: How to include dependencies, like additional JavaScript libraries, in your project? SBT standardizes on WebJars, so you are supposed to use WebJars to manage your dependencies. Unfortunately, in my experience it proved to be inadequate.

The Problem with WebJars

As mentioned, the standard Scala.js way to retrieve JavaScript dependencies is based on WebJars. Scala is, after all, a JVM language. Scala.js uses SBT for building programs primarily, and finally SBT is excellent in managing JAR dependencies.

For this reason, the WebJar format was defined exactly for importing JavaScript dependencies in the JVM world. A WebJar is a JAR file including web assets, where simple JAR file only includes compiled Java classes. So, the idea of Scala.js is you should import your JavaScript dependencies by simply adding WebJar dependencies, in a similar way Scala is adding JAR dependencies.

Nice Idea, Except It Doesn’t Work

The biggest problem with Webjars is that an arbitrary version on a random JavaScript library is rarely available as a WebJar. At the same time, the vast majority of JavaScript libraries are available as NPM modules. However, there is a supposed bridge between NPM and WebJars with an automated npm-to-webjar packager. So I tried to import a library, available as an NPM module. In my case, it was VoxelJS, a library for building Minecraft-like worlds in a web page. I tried to request the library as a WebJar, but the bridge failed simply because there are no license fields in the descriptor.

You may also face this frustrating experience for other reasons with other libraries. Simply put, it looks like you cannot access each library in the wild as a WebJar. The requirement you have to use WebJars for accessing JavaScript libraries appears to be too limiting.

Enter NPM and Browserify

As I already pointed out, the standard packaging format for most of the JavaScript libraries is the Node Package Manager, or NPM, included in any version of Node.js. By using NPM, you can easily access almost all the available JavaScript libraries.

Note that NPM is Node package manager. Node is a server-side implementation of the V8 JavaScript engine, and it installs packages to be used on the server side by Node.js. As is, NPM is useless for the browser. However, the usefulness of NPM has been extended a while to work with browser applications, thanks to the ubiquitous Browserify tool, which is of course also distributed as an NPM package.

Browserify is, simply put, a packager for the browser. It will collect NPM modules producing a “bundle” usable in a browser application. Many JavaScript developers work this way - they manage packages with NPM, and afterward Browserify them to use in the web application. Mind that there are other tools that work in the same way, like Webpack.

Filling the Gap from SBT to NPM

For the reasons I just described, what I wanted was a way to install dependencies from the web with NPM, invoke Browserify to gather dependencies for the browser, and then use them with Scala.js. The task turned out to be a bit more complicated than I expected, but still possible. Indeed, I did the job and I am describing it here.

For simplicity, I chose Browserify also because I found it’s possible to run it within SBT. I have not tried with Webpack, although I guess it is possible, too. Luckily, I didn’t have to start in a vacuum. There are a number of pieces already in place:

• SBT already supports NPM. The sbt-web plugin, developed for the Play Framework, can install NPM dependencies.
• SBT supports execution of JavaScript. You can execute Node tools without installing Node itself, thanks to the sbt-jsengine plugin.
• Scala.js can use a generated bundle. In Scala.js, there is a concatenation function to include in your application arbitrary JavaScript libraries.

Using these features, I created an SBT task that can download the NPM dependencies and then invoke Browserify, producing a bundle.js file. I tried to integrate the procedure in the compilation chain, and I can run the whole thing automatically, but having to process the bundling at each compilation is simply too slow. Also, you do not change dependencies all the time; hence, it is reasonable you have to create manually a bundle once a while when you change dependencies.

So, my solution was to build a subproject. This subproject downloads and packages JavaScript libraries with NPM and Browserify. Then, I added a bundle command to perform the gathering of dependencies. The resulting bundle is added to resources to be used in the Scala.js application.

You are supposed to execute this “bundle” manually whenever you change your JavaScript dependencies. As mentioned, it is not automated in the compilation chain.

How to Use the Bundler

If you want to use my example, do the following: first, checkout the repository with the usual Git command.

git clone https://github.com/sciabarra/scalajs-browserify/

Then, copy the bundle folder in your Scala.js project. It is a subproject for bundling. To connect to the main project, you should add the following lines in your build.sbt file:

val bundle = project.in(file("bundle"))

jsDependencies += ProvidedJS / "bundle.js"

addCommandAlias("bundle", "bundle/bundle")

Also, you have to add the following lines to your project/plugins.sbt file:

addSbtPlugin("com.typesafe.sbt" % "sbt-web" % "1.1.1")

addSbtPlugin("com.typesafe.sbt" % "sbt-js-engine" % "1.1.3")

Once done, you have a new command, bundle, that you can use to gather your dependencies. It will generate a file bundle.js under your src/main/resources folder.

How Is the Bundle Included in Your Scala.js Application?

The bundle command just described gathers dependencies with NPM and then creates a bundle.js. When you run the fastOptJS or fullOptJS commands, ScalaJS will create a myproject-jsdeps.js, including all the resources you specified as a JavaScript dependency, hence also your bundle.js. To include bundled dependencies in your application, you are supposed to use the following inclusions:

<script src="target/scala-2.11/myproject-jsdeps.js"></script>
<script src="target/scala-2.11/myproject-fastopt.js"></script>
<script src="target/scala-2.11/myproject-launcher.js"></script>

Your bundle is now available as part of the myproject-jsdeps.js. The bundle is ready, and we have somewhat finished our task (importing dependencies and exporting them to the browser). Next step is using the JavaScript libraries, which is a different problem, a Scala.js coding problem. For completeness, we will now discuss how to use the bundle in Scala.js and create facades to use the libraries we imported.

Using a Generic JavaScript Library in Your Scala.js Application

To recap, we have just seen how to use NPM and Browserify to create a bundle, and include that bundle in Scala.js. But how can we use a generic JavaScript library?

The complete process, that we will explain in detail in the rest of the post, is:

• Select your libraries from the NPM and include them in the bundle/package.json.
• Load them with require in a library module file, in bundle/lib.js.
• Write Scala.js facades to interpret the Bundle object in Scala.js.
• Finally, code your application using the newly typed libraries.

Adding a Dependency

Using NPM, you have to include your dependencies in the package.json file, which is the standard.

So, let’s assume you want to use two famous libraries like jQuery and Loadash. This example is only for demonstration purposes, because there is already an excellent wrapper for jQuery available as a dependency for Scala.js with a proper module, and Lodash is useless in the Scala world. Nonetheless, I think it is a good example, but take it only as an example.

So, go to the npmjs.com website and locate the library you want to use and select a version too. Let’s suppose you choose jquery-browserify version 13.0.0 and lodash version 4.3.0. Then, update the dependencies block of your packages.json as follows:

"dependencies": {
  "browserify": "13.0.0",
  "jquery-browserify": "1.8.1",
  "lodash": "4.3.0"
}

Always keep browserify, as it is needed to generate the bundle. You are not required to include it in the bundle, though.

Note that if you have NPM installed, you can just type from the bundle directory:

npm install --save jquery-browserify  lodash

It will also update the package.json. If you do not have NPM installed, do not worry. SBT will install a Java version of Node.js and NPM, download the required JARs and run them. All of this is managed when you run the bundle command from SBT.

Exporting the Library

Now we know how to download the packages. Next step is to instruct Browserify to gather them in a bundle and make them available to the rest of the application.

Browserify is a gatherer of require, emulating the Node.js behavior for the browser, which means you need to have somewhere a require importing your library. Because we need to export those libraries to Scala.js, the bundle is also generating a top level JavaScript object named Bundle. So, what you need to do is to edit the lib.js, which exports a JavaScript object, and require all your libraries as fields of this object.

If we want to export to Scala.js jQuery and Lodash libraries, in code this means:

module.exports = {
 "jquery": require("jquery-browserify"),
 "lodash": require("lodash")
}

Now, just execute the command bundle and the library will be downloaded, gathered and placed in the bundle, ready to be used in your Scala.js application.

Accessing the Bundle

So far:

• You installed the bundle subproject in your Scala.js project and configured it correctly.
• For any library you wanted, you added it in the package.json.
• You required them in the lib.js.
• You executed the bundle command.

As a result, you have now a Bundle top level JavaScript object, providing all the entry points for the libraries, available as fields of this object.

Now you are ready to use it with Scala.js. In the simplest case you can do something like this to access the libraries:

@js.native
object Bundle extends js.Object {
  def jquery : js.Any = js.native
  def lodash: js.Any = js.native
}

This code lets you access the libraries from Scala.js. However, it is not the way you should do it with Scala.js, because the libraries are still untyped. Instead, you should write typed “facades”, or wrappers, so you can use the originally untyped JavaScript libraries in a typed scalish way.

I can not tell you here how to write the facades since it depends on the specific JavaScript library you want to wrap in Scala.js. I will only show an example, to complete the discussion. Please check the official Scala.js documentation for more details. Also, you can consult the list of available facades and read the source code for inspiration.

So far, we covered the process for arbitrary, still unmapped libraries. In the rest of the article I am referring to library with an already available facade, so the discussion is only an example.

Wrapping a JavaScript API in Scala.js

When you use require in JavaScript, you get an object that can be many different things. It can be a function or an object. It can even be just a string or a boolean, in which case the require is invoked only for the side effects.

In the example I am doing, jquery is a function, returning an object which provides additional methods. Typical usage is jquery(<selector>).<method>. This is a simplification, because jquery also allows for $.<method> usage, but in this simplified example I am not going to cover all those cases. Note in general, for complex JavaScript libraries not all of the API can be easily mapped to static Scala types. You may need to resort to js.Dynamic providing a dynamic (untyped) interface to JavaScript object.

So, to capture just the more common use case, I defined in the Bundle object, jquery:

def jquery : js.Function1[js.Any, Jquery] = js.native

This function will return a jQuery object. An instance of a trait is in my case defined with a single method (a simplification, you can add your own):

@js.native
trait Jquery extends js.Object {
   def text(arg: js.Any): Jquery = js.native
}

For the Lodash library, we model the whole library as a JavaScript object since it is a collection of functions you can call directly:

def lodash: Lodash = js.native  

Where the lodash trait is just as follows (also a simplification, you can add your methods here):

@js.native
trait Lodash extends js.Object {
   def camelCase(arg: js.Any): String = js.native
}

Using those definitions, we can now finally write Scala code using the underlying jQuery and Lodash libraries, both loaded from NPM and then browseried:

object Main extends JSApp {
  def main(): Unit = {
    import Bundle._
    jquery("#title").text(lodash.camelCase("This is a test"))
  }
}

You can check the complete example here.

Conclusion

I am a Scala developer, and I was excited when I discovered Scala.js because I could use the same language for both the server and the client. Since Scala is more similar to JavaScript than it is Java, Scala.js is pretty easy and natural in the browser. Furthermore, you can also use powerful features of Scala as a wealthy collection of libraries, macros, and powerful IDEs and build tools. Other key advantages are that you can share code between server and client. There are plenty of cases where this feature is useful. If you use a transpiler for Javascript like Coffeescript, Babel or Typescript, you will not notice too many differences when using Scala.js, but still there are many advantages. The secret is to take the best of each world and ensure they work together nicely.