Declarative programming is, currently, the dominant paradigm of an extensive and diverse set of domains such as databases, templating and configuration management.

In a nutshell, declarative programming consists of instructing a program on what needs to be done, instead of telling it how to do it. In practice, this approach entails providing a domain-specific language (DSL) for expressing what the user wants, and shielding them from the low-level constructs (loops, conditionals, assignments) that materialize the desired end state.

While this paradigm is a remarkable improvement over the imperative approach that it replaced, I contend that declarative programming has significant limitations, limitations that I explore in this article. Moreover, I propose a dual approach that captures the benefits of declarative programming while superseding its limitations.

CAVEAT: This article emerged as the result of a multi-year personal struggle with declarative tools. Many of the claims I present here are not thoroughly proven, and some are even presented at face value. A proper critique of declarative programming would take considerable time, effort, and I would have to go back and use many of these tools; my heart is not in such an undertaking. The purpose of this article is to share a few thoughts with you, pulling no punches, and showing what worked for me. If you’ve struggled with declarative programming tools, you might find respite and alternatives. And if you enjoy the paradigm and its tools, don’t take me too seriously.

If declarative programming works well for you, I’m in no position to tell you otherwise.

You can love or hate declarative programming, but you cannot afford to ignore it.

You can love or hate declarative programming, but you cannot afford to ignore it.

The Merits Of Declarative Programming

Before we explore the limits of declarative programming, it is necessary to understand its merits.

Arguably the most successful declarative programming tool is the relational database (RDB). It might even be the first declarative tool. In any case, RDBs exhibit the two properties that I consider archetypical of declarative programming:

  • A domain specific language (DSL): the universal interface for relational databases is a DSL named Structured Query Language, most commonly known as SQL.
  • The DSL hides the lower level layer from the user: ever since Edgar F. Codd’s original paper on RDBs, it is plain that the power of this model is to dissociate the desired queries from the underlying loops, indexes and access paths that implement them.

Before RDBs, most database systems were accessed through imperative code, which is heavily dependent on low-level details such as the order of records, indexes and the physical paths to the data itself. Because these elements change over time, code often stops working because of some underlying change in the structure of the data. The resulting code is hard to write, hard to debug, hard to read and hard to maintain. I’ll go out a limb and say that most of this code was in, all likelihood, long, full of proverbial rats’ nests of conditionals, repetition and subtle, state-dependent bugs.

In the face of this, RDBs provided a tremendous productivity leap for systems developers. Now, instead of thousands of lines of imperative code, you had a clearly defined data scheme, plus hundreds (or even just tens) of queries. As a result, applications had only to deal with an abstract, meaningful and lasting representation of data, and interface it through a powerful, yet simple query language. The RDB probably raised the productivity of programmers, and companies that employed them, by an order of magnitude.

What are the commonly listed advantages of declarative programming?

Proponents of declarative programming are quick to point out the advantages. However, even they admit it comes with trade-offs.

Proponents of declarative programming are quick to point out the advantages. However, even they admit it comes with trade-offs.
  1. Readability/usability: a DSL is usually closer to a natural language (like English) than to pseudocode, hence more readable and also easier to learn by non-programmers.
  2. Succinctness: much of the boilerplate is abstracted by the DSL, leaving less lines to do the same work.
  3. Reuse: it is easier to create code that can be used for different purposes; something that’s notoriously hard when using imperative constructs.
  4. Idempotence: you can work with end states and let the program figure it out for you. For example, through an upsert operation, you can either insert a row if it is not there, or modify it if it is already there, instead of writing code to deal with both cases.
  5. Error recovery: it is easy to specify a construct that will stop at the first error instead of having to add error listeners for every possible error. (If you’ve ever written three nested callbacks in node.js, you know what I mean.)
  6. Referential transparency: although this advantage is commonly associated with functional programming, it is actually valid for any approach that minimizes manual handling of state and relies on side effects.
  7. Commutativity: the possibility of expressing an end state without having to specify the actual order in which it will be implemented.

While the above are all commonly cited advantages of declarative programming, I would like to condense them into two qualities, which will serve as guiding principles when I propose an alternative approach.

  1. A high-level layer tailored to a specific domain: declarative programming creates a high-level layer using the information of the domain to which it applies. It is clear that if we’re dealing with databases, we want a set of operations for dealing with data. Most of the seven advantages above stem from the creation of a high-level layer that is precisely tailored to a specific problem domain.
  2. Poka-yoke (fool-proofness): a domain-tailored high-level layer hides the imperative details of the implementation. This means that you commit far fewer errors because the low-level details of the system are simply not accessible. This limitation eliminates many classes of errors from your code.

Two Problems With Declarative Programming

In the following two sections, I will present the two main problems of declarative programming: separateness and lack of unfolding. Every critique needs its bogeyman, so I will use HTML templating systems as a concrete example of the shortcomings of declarative programming.

The Problem With DSLs: Separateness

Imagine that you need to write a web application with a non-trivial number of views. Hard coding these views into a set of HTML files is not an option because many components of these pages change.

The most straightforward solution, which is to generate HTML by concatenating strings, seems so horrible that you will quickly look for an alternative. The standard solution is to use a template system. Although there are different types of template systems, we will sidestep their differences for the purpose of this analysis. We can consider all of them to be similar in that the main mission of template systems is to provide an alternative to code that concatenates HTML strings using conditionals and loops, much like RDBs emerged as an alternative to code that looped through data records.

Let’s suppose we go with a standard templating system; you will encounter three sources of friction, which I will list in ascending order of importance. The first is that the template necessarily resides in a file separate from your code. Because the templating system uses a DSL, the syntax is different, so it cannot be in the same file. In simple projects, where file counts are low, the need to keep separate template files may duplicate or treble the amount of files.

I open an exception for Embedded Ruby templates (ERB), because those are integrated into Ruby source code. This is not the case for ERB-inspired tools written in other languages since those templates must also be stored as different files.

The second source of friction is that the DSL has its own syntax, one different from that of your programming language. Hence, modifying the DSL (let alone writing your own) is considerably harder. To go under the hood and change the tool, you need to learn about tokenizing and parsing, which is interesting and challenging, but hard. I happen to see this as a disadvantage.

How can one vizualise a DSL? It’s not easy, but let’s just say a DSL is a clean, shiny layer on top of low-level constructs.

How can one vizualise a DSL? It’s not easy, but let’s just say a DSL is a clean, shiny layer on top of low-level constructs.

You may ask, “Why on earth would you want to modify your tool? If you are doing a standard project, a well-written standard tool should fit the bill.” Maybe yes, maybe no.

A DSL never has the full power of a programming language. If it did, it wouldn’t be a DSL anymore, but rather a full programming language.

But isn’t that the whole point of a DSL? To not have the full power of a programming language available, so that we can achieve abstraction and eliminate most sources of bugs? Maybe, yes. However, most DSLs start simple and then gradually incorporate a growing number of the facilities of a programming language until, in fact, it becomes one. Template systems are a perfect example. Let’s see the standard features of template systems and how they correlate to programming language facilities:

  • Replace text within a template: variable substitution.
  • Repetition of a template: loops.
  • Avoid printing a template if a condition is not met: conditionals.
  • Partials: subroutines.
  • Helpers: subroutines (the only difference with partials is that helpers can access the underlying programming language and let you out of the DSL straightjacket).

This argument, that a DSL is limited because it simultaneously covets and rejects the power of a programming language, is directly proportional to the extent that the features of the DSL are directly mappable to the features of a programming language. In the case of SQL, the argument is weak because most of the things SQL offers are nothing like what you find in a normal programming language. At the other end of the spectrum, we find template systems where virtually every feature is making the DSL converge towards BASIC.

Let’s now step back and contemplate these three quintessential sources of friction, summed up by the concept of separateness. Because it is separate, a DSL needs to be located on a separate file; it is harder to modify (and even harder to write your own), and (often, but not always) needs you to add, one by one, the features you miss from a real programming language.

Separateness is an inherent problem of any DSL, no matter how well designed.

We now turn to a second problem of declarative tools, which is widespread but not inherent.

Another Problem: Lack Of Unfolding Leads To Complexity

If I had written this article a few months ago, this section would have been named Most Declarative Tools Are #@!$#@! Complex But I Don’t Know Why. In the process of writing this article I found a better way of putting it: Most Declarative Tools Are Way More Complex Than They Need To Be. I will spend the rest of this section explaining why. To analyze the complexity of a tool, I propose a measure called the complexity gap. The complexity gap is the difference between solving a given problem with a tool versus solving it in the lower level (presumably, plain imperative code) that the tool intends to replace. When the former solution is more complex than the latter, we are in presence of the complexity gap. By more complex, I mean more lines of code, code that’s harder to read, harder to modify and harder to maintain, but not necessarily all of these at the same time.

Please note that we’re not comparing the lower level solution against the best possible tool, but rather against no tool. This echoes the medical principle of “First, do no harm”.

Signs of a tool with a large complexity gap are:

  • Something that takes a few minutes to describe in rich detail in imperative terms will take hours to code using the tool, even when you know how to use the tool.
  • You feel you are constantly working around the tool rather than with the tool.
  • You are struggling to solve a straightforward problem that squarely belongs in the domain of the tool you are using, but the best Stack Overflow answer you find describes a workaround.
  • When this very straightforward problem could be solved by a certain feature (which does not exist in the tool) and you see a Github issue in the library that features a long discussion of said feature with +1s interspersed.
  • A chronic, itching, longing to ditch the tool and do the whole thing yourself inside a _ for- loop_.

I might have fallen prey to emotion here since template systems are not that complex, but this comparatively small complexity gap is not a merit of their design, but rather because the domain of applicability is quite simple (remember, we’re just generating HTML here). Whenever the same approach is used for a more complex domain (such as configuration management) the complexity gap may quickly turn your project into a quagmire.

That said, it is not necessarily unacceptable for a tool to be somewhat more complex than the lower level it intends to replace; if the tool yields code that is more readable, concise and correct, it can be worth it t. It’s an issue when the tool is several times more complex than the problem it replaces; this is flat-out unacceptable. Brian Kernighan famously stated that, “Controlling complexity is the essence of computer programming.” If a tool adds significant complexity to your project, why even use it?

The question is, why are some declarative tools so much more complex than they need be? I think it would be a mistake to blame it on poor design. Such a general explanation, a blanket ad-hominem attack on the authors of these tools, is not fair. There has to be a more accurate and enlightening explanation.

Origami time! A tool with a high-level interface to an abstract lower level has to unfold the higher level from the lower one.

Origami time! Origami time! A tool with a high-level interface to an abstract lower level has to unfold the higher level from the lower one.

My contention is that any tool that offers a high level interface to abstract a lower level must unfold this higher level from the lower one. The concept of unfolding comes from Christopher Alexander’s magnum opus, The Nature of Order - in particular Volume II. It is (hopelessly) beyond the scope of this article (not to mention my understanding) to summarize the implications of this monumental work for software design; I believe its impact will be huge in years to come. It is also beyond this article to provide a rigorous definition of unfolding processes. I will use here the concept in a heuristic way.

An unfolding process is one that, in a stepwise fashion, creates further structure without negating the existing one. At every step, each change (or differentiation, to use Alexander’s term) remains in harmony with any previous structure, when previous structure is, simply, a crystallized sequence of past changes.

Interestingly enough, Unix is a great example of the unfolding of a higher level from a lower one. In Unix, two complex features of the operative system, batch jobs and coroutines (pipes), are simply extensions of basic commands. Because of certain fundamental design decisions, such as making everything a stream of bytes, the shell being a userland program and standard I/O files, Unix is able to provide these sophisticated features with minimal complexity.

To underline why these are excellent examples of unfolding, I would like to quote a few excerpts of a 1979 paper by Dennis Ritchie, one of the authors of Unix:

On batch jobs:

… the new process control scheme instantly rendered some very valuable features trivial to implement; for example detached processes (with &) and recursive use of the shell as a command. Most systems have to supply some sort of special batch job submission facility and a special command interpreter for files distinct from the one used interactively.

On coroutines:

The genius of the Unix pipeline is precisely that it is constructed from the very same commands used constantly in simplex fashion.

UNIX pioneers Dennis Ritchie and Ken Thompson created a powerful demonstration of unfolding in their OS. They also saved us from a dystopian all-Windows future.

UNIX pioneers Dennis Ritchie and Ken Thompson created a powerful demonstration of unfolding in their OS. They also saved us from a dystopian all-Windows future.

This elegance and simplicity, I argue, comes from an unfolding process. Batch jobs and coroutines are unfolded from previous structures (commands run in a userland shell). I believe that because of the minimalist philosophy and limited resources of the team that created Unix, the system evolved stepwise, and as such, was able to incorporate advanced features without turning its back on to the basic ones because there weren’t enough resources to do otherwise.

In the absence of an unfolding process, the high level will be considerably more complex than necessary. In other words, the complexity of most declarative tools stem from the fact that their high level does not unfold from the low level they intend to replace.

This lack of unfoldance, if you forgive the neologism, is routinely justified by the necessity to shield the user from the lower level. This emphasis on poka-yoke (protecting the user from low level errors) comes at the expense of a large complexity gap that is self-defeating because the extra complexity will generate new classes of errors. To add insult to injury, these classes of errors have nothing to do with the problem domain but rather with the tool itself. We would not go too far if we describe these errors as iatrogenic.

Declarative templating tools, at least when applied to the task of generating HTML views, are an archetypical case of a high level that turns its back on the low level it intends to replace. How so? Because generating any non-trivial view requires logic, and templating systems, especially logic-less ones, banish logic through the main door and then smuggle some of it back through the cat door.

Note: An even weaker justification for a large complexity gap is when a tool is marketed as magic, or something that just works, the opaqueness of the low level is supposed to be an asset because a magic tool is always supposed to work without you understanding why or how. In my experience, the more magical a tool purports to be, the faster it transmutes my enthusiasm into frustration.

But what about the separation of concerns? Shouldn’t view and logic remain separate? The core mistake, here, is to put business logic and presentation logic in the same bag. Business logic certainly has no place in a template, but presentation logic exists nevertheless. Excluding logic from templates pushes presentation logic into the server where it is awkwardly accommodated. I owe the clear formulation of this point to Alexei Boronine, who makes an excellent case for it in this article.

My feeling is that roughly two thirds of the work of a template resides in its presentation logic, while the other third deals with generic issues such as concatenating strings, closing tags, escaping special characters, and so on. This is the two-faced low level nature of generating HTML views. Templating systems deal appropriately with the second half, but they don’t fare well with the first. Logic-less templates flat out turn their back on this problem, forcing you to solve it awkwardly. Other template systems suffer because they truly need to provide a non-trivial programming language so their users can actually write presentation logic.

To sum up; declarative templating tools suffer because:

  • If they were to unfold from their problem domain, they would have to provide ways to generate logical patterns;
  • A DSL that provides logic is not really a DSL, but a programming language. Note that other domains, like configuration management, also suffer from lack of “unfoldance.”

I would like to close the critique with an argument that is logically disconnected from the thread of this article, but deeply resonates with its emotional core: We have limited time to learn. Life is short, and on top of that, we need to work. In the face of our limitations, we need to spend our time learning things that will be useful and withstand time, even in the face of fast changing technology. That is why I exhort you to use tools that don’t just provide a solution but actually shed a bright light on the domain of its own applicability. RDBs teach you about data, and Unix teaches you about OS concepts, but with unsatisfactory tools that don’t unfold, I’ve always felt I was learning the intricacies of a sub-optimal solution while remaining in the dark about the nature of problem it intends to solve.

The heuristic I suggest you to consider is, value tools that illuminate their problem domain, instead of tools that obscure their problem domain behind purported features.

The Twin Approach

To overcome the two problems of declarative programming, which I have presented here, I propose a twin approach:

  • Use a data structure domain specific language (dsDSL), to overcome separateness.
  • Create a high level that unfolds from the lower level, to overcome the complexity gap.

dsDSL

A data structure DSL (dsDSL) is a DSL that is built with the data structures of a programming language. The core idea is to use the basic data structures you have available, such as strings, numbers, arrays, objects and functions, and combine them to create abstractions to deal with a specific domain.

We want to keep the power of declaring structures or actions (high level) without having to specify the patterns that implement these constructs (low level). We want to overcome the separateness between the DSL and our programming language so that we are free to use the full power of a programming language whenever we need it. This is not only possible but straightforward through dsDSLs.

If you asked me a year ago, I would have thought that the concept of dsDSL was novel, then one day, I realized that JSON itself was a perfect example of this approach! A parsed JSON object consists of data structures that declaratively represent data entries in order to get the advantages of the DSL while also making it easy to parse and handle from within a programming language. (There might be other dsDSLs out there, but so far I haven’t come across any. If you know of one, I would really appreciate your mentioning it in the comments section.)

Like JSON, a dsDSL has the following attributes:

  1. It consists of a very small set of functions: JSON has two main functions, parse and stringify.
  2. Its functions most commonly receive complex and recursive arguments: a parsed JSON is an array, or object, which usually contains further arrays and objects inside.
  3. The inputs to these functions conform to very specific forms: JSON has an explicit and strictly enforced validation schema to tell valid from invalid structures.
  4. Both the inputs and the outputs of these functions can be contained and generated by a programming language without a separate syntax.

But dsDSLs go beyond JSON in many ways. Let’s create a dsDSL for generating HTML using Javascript. Later I will touch on the issue of whether this approach may be extended to other languages (spoiler: It can definitely be done in Ruby and Python, but probably not in C).

HTML is a markup language composed of tags delimited by angle brackets (< and >). These tags may have optional attributes and contents. Attributes are simply a list of key/value attributes, and contents may be either text or other tags. Both attributes and contents are optionals for any given tag. I’m simplifying somewhat, but it is accurate.

A straightforward way to represent an HTML tag in a dsDSL is by using an array with three elements: - Tag: a string. - Attributes: an object (of the plain, key/value type) or undefined (if no attributes are necessary). - Contents: a string (text), an array (another tag) or undefined (if there’s no contents).

For example, <a href="views">Index</a> can be written as ['a', {href: 'views'}, 'Index'].

If we want to embed this anchor element into a div with class links, we can write: ['div', {class: 'links'}, ['a', {href: 'views'}, 'Index']].

To list several html tags at the same level, we can wrap them in an array:

[
   ['h1', 'Hello!'],
   ['a', {href: 'views'}, 'Index']
]

The same principle may be applied to creating multiple tags within a tag:

['body', [
   ['h1', 'Hello!'],
   ['a', {href: 'views'}, 'Index']
]]

Of course, this dsDSL won’t get us far if we don’t generate HTML from it. We need a generate function which will take our dsDSL and yield a string with HTML. So if we run generate (['a', {href: 'views'}, 'Index']), we will get the string <a href="views">Index</a>.

The idea behind any DSL is to specify a few constructs with a specific structure which is then passed to a function. In this case, the structure that makes up the dsDSL is this array, which has one to three elements; these arrays have a specific structure. If generate thoroughly validates its input (and it is both easy and important to thoroughly validate input, since these validation rules are the precise analog of a DSL’s syntax), it will tell you exactly where you went wrong with your input. After a while, you’ll start to recognize what distinguishes a valid structure in a dsDSL, and this structure will be highly suggestive of the underlying thing it generates.

Now, what are the merits of a dsDSL in contraposition to a DSL?

  • A dsDSL is an integral part of your code. It leads to lower line counts, file counts, and an overall reduction of overhead.
  • dsDSLs are easy to parse (hence easier to implement and modify). Parsing is merely iterating through the elements of an array or object. Likewise, dsDSLs are comparatively easy to design because instead of creating a new syntax (that everybody will hate) you can stick with the syntax of your programming language (which everybody hates but at least they already know it).
  • A dsDSL has all the power of a programming language. This means that a dsDSL, when properly employed, has the advantage of both a high and a low level tool.

Now, the last claim is a strong one, so I’m going to spend the rest of this section supporting it. What do I mean by properly employed? To see this in action, let’s consider an example in which we want to construct a table to display the information from an array named DATA.

var DATA = [
   {id: 1, description: 'Product 1', price: 20,  onSale: true,  categories: ['a']},
   {id: 2, description: 'Product 2', price: 60,  onSale: false, categories: ['b']},
   {id: 3, description: 'Product 3', price: 120, onSale: false, categories: ['a', 'c']},
   {id: 4, description: 'Product 4', price: 45,  onSale: true,  categories: ['a', 'b']}
]

In a real application, DATA will be generated dynamically from a database query.

Moreover, we have a FILTER variable which, when initialized, will be an array with the categories we want to display.

We want our table to:

  • Display table headers.
  • For each product, show the fields: description, price and categories.
  • Don’t print the id field, but add it as an id attribute for each row. ALTERNATE VERSION: Add an id attribute to each tr element.
  • Place a class onSale if the product is on sale.
  • Sort the products by descending price.
  • Filter certain products by category. If FILTER is an empty array, we will display all products. Otherwise, we will only display the products where the category of the product is contained within FILTER.

We can create the presentation logic that matches this requirement in ~20 lines of code:

function drawTable (DATA, FILTER) {

   var printableFields = ['description', 'price', 'categories'];

   DATA.sort (function (a, b) {return a.price - b.price});

   return ['table', [
      ['tr', dale.do (printableFields, function (field) {
         return ['th', field];
      })],
      dale.do (DATA, function (product) {
         var matches = (! FILTER || FILTER.length === 0) || dale.stop (product.categories, true, function (category) {
            return FILTER.indexOf (category) !== -1;
         });

         return matches === false ? [] : ['tr', {
            id: product.id,
            class: product.onSale ? 'onsale' : undefined
         }, dale.do (printableFields, function (field) {
            return ['td', product [field]];
         })];
      })
   ]];
}

I concede this is not a straightforward example, however, it represents a fairly simple view of the four basic functions of persistent storage, also known as CRUD. Any non-trivial web application will have views that are more complex than this.

Let’s now see what this code is doing. First, it defines a function, drawTable, to contain the presentation logic of drawing the product table. This function receives DATA and FILTER as parameters, so it can be used for different data sets and filters. drawTable fulfills the double role of partial and helper.

   var drawTable = function (DATA, FILTER) {

The inner variable, printableFields, is the only place where you need to specify which fields are printable ones, avoiding repetition and inconsistencies in the face of changing requirements.

   var printableFields = ['description', 'price', 'categories'];

We then sort DATA according to the price of its products. Notice that different and more complex sort criteria would be straightforward to implement since we have the entire programming language at our disposal.

   DATA.sort (function (a, b) {return a.price - b.price});

Here we return an object literal; an array which contains table as its first element and its contents as the second. This is the dsDSL representation of the <table> we want to create.

   return ['table', [

We now create a row with the table headers. To create its contents, we use dale.do which is a function like Array.map, but one that also works for objects. We will iterate printableFields and generate table headers for each of them:

      ['tr', dale.do (printableFields, function (field) {
         return ['th', field];
      })],

Notice that we have just implemented iteration, the workhorse of HTML generation, and we didn’t need any DSL constructs; we only needed a function to iterate a data structure and return dsDSLs. A similar native, or user-implemented function, would have done the trick as well.

Now iterate through the products contained in DATA.

      dale.do (DATA, function (product) {

We check whether this product is left out by FILTER. If FILTER is empty, we will print the product. If FILTER is not empty, we will iterate through the categories of the product until we find one that is contained within FILTER. We do this using dale.stop.

         var matches = (! FILTER || FILTER.length === 0) || dale.stop (product.categories, true, function (category) {
            return FILTER.indexOf (category) !== -1;
         });

Notice the intricacy of the conditional; it is precisely tailored to our requirement and we have total freedom for expressing it because we are in a programming language rather than a DSL.

If matches is false, we return an empty array (so we don’t print this product). Otherwise, we return a <tr> with its proper id and class and we iterate through printableFields to, well, print the fields.


         return matches === false ? [] : ['tr', {
            id: product.id,
            class: product.onSale ? 'onsale' : undefined
         }, dale.do (printableFields, function (field) {
            return ['td', product [field]];

Of course we close everything that we opened. Isn’t syntax fun?

         })];
      })
   ]];
}

Now, how do we incorporate this table into a wider context? We write a function named drawAll that will invoke all functions that generate the views. Apart from drawTable, we might also have drawHeader, drawFooter and other comparable functions, all of which will return dsDSLs.

var drawAll = function () {
   return generate ([
      drawHeader (),
      drawTable (DATA, FILTER),
      drawFooter ()
   ]);
}

If you don’t like how the above code looks, nothing I say will convince you. This is a dsDSL at its best. You might as well stop reading the article (and drop a mean comment too because you’ve earned the right to do so if you’ve made it this far!). But seriously, if the code above doesn’t strike you as elegant, nothing else in this article will.

For those who are still with me, I would like to go back to the main claim of this section, which is that a dsDSL has the advantages of both the high and the low level:

  • The advantage of the low level resides in writing code whenever we want, getting out of the straightjacket of the DSL.
  • The advantage of the high level resides in using literals that represent what we want to declare and letting the functions of the tool convert that into the desired end state (in this case, a string with HTML).

But how is this truly different from purely imperative code? I think ultimately the elegance of the dsDSL approach boils down to the fact that code written in this way mostly consists of expressions, instead of statements. More precisely, code that uses a dsDSL is almost entirely composed of:

  • Literals that map to lower level structures.
  • Function invocations or lambdas within those literal structures that return structures of the same kind.

Code that consists mostly of expressions and which encapsulate most statements within functions is extremely succinct because all patterns of repetition can be easily abstracted. You can write arbitrary code as long as that code returns a literal that conforms to a very specific, non-arbitrary form.

A further characteristic of dsDSLs (which we don’t have time to explore here) is the possibility of using types to increase the richness and succinctness of the literal structures. I will expound on this issue on a future article.

Might it be possible to create dsDSLs beyond Javascript, the One True Language? I think that it is, indeed, possible, as long as the language supports:

  • Literals for: arrays, objects (associative arrays), function invocations, and lambdas.
  • Runtime type detection
  • Polymorphism and dynamic return types

I think this means that dsDSLs are tenable in any modern dynamic language (i.e.: Ruby, Python, Perl, PHP), but probably not in C or Java.

Walk, Then Slide: How To Unfold The High From The Low

In this section I will attempt to show a way for unfolding a high level tool from its domain. In a nutshell, the approach consists of the following steps

  1. Take two to four problems that are representative instances of a problem domain. These problems should be real. Unfolding the high level from the low one is a problem of induction, so you need real data to come up with representative solutions.
  2. Solve the problems with no tool in the most straightforward way possible.
  3. Stand back, take a good look at your solutions, and notice the common patterns among them.
  4. Find the patterns of representation (high level).
  5. Find the patterns of generation (low level).
  6. Solve the same problems with your high level layer and verify that the solutions are indeed correct.
  7. If you feel that you can easily represent all the problems with your patterns of representation, and the generation patterns for each of these instances produce correct implementations, you’re done. Otherwise, go back to the drawing board.
  8. If new problems appear, solve them with the tool and modify it accordingly.
  9. The tool should converge asymptotically to a finished state, no matter how many problems it solves. In other words, the complexity of the tool should remain constant, rather than growing with the amount of problems it solves.

Now, what the hell are patterns of representation and patterns of generation? I’m glad you asked. The patterns of representation are the patterns in which you should be able to express a problem that belongs to the domain that concerns your tool. It is an alphabet of structures that allows you to write any pattern you might wish to express within its domain of applicability. In a DSL, these would be the production rules. Let’s go back to our dsDSL for generating HTML.

The humble HTML tag is a good example of patterns of representation. Let’s take a closer look at these basic patterns.

The humble HTML tag is a good example of patterns of representation. Let’s take a closer look at these basic patterns.

The patterns of representation for HTML are the following:

  • A single tag: ['TAG']
  • A single tag with attributes: ['TAG', {attribute1: value1, attribute2: value2, ...}]
  • A single tag with contents: ['TAG', 'CONTENTS']
  • A single tag with both attributes and contents: ['TAG', {attribute1: value1, ...}, 'CONTENTS']
  • A single tag with another tag inside: ['TAG1', ['TAG2', ...]]
  • A group of tags (standalone or inside another tag): [['TAG1', ...], ['TAG2', ...]]
  • Depending on a condition, place a tag or no tag: condition ? ['TAG', ...] : [] / Depending on a condition, place an attribute or no attribute: ['TAG', {class: condition ? 'someClass': undefined}, ...]

These instances can be represented with the dsDSL notation we determined in the previous section. And this is all you need to represent any HTML you might need. More sophisticated patterns, such as conditional iteration through an object to generate a table, may be implemented with functions that return the patterns of representation above, and these patterns map directly to HTML tags.

If the patterns of representation are the structures you use to express what you want, the patterns of generation are the structures your tool will use to convert patterns of representation into the lower level structures. For HTML, these are the following:

  • Validate the input (this is actually is an universal pattern of generation).
  • Open and close tags (but not the void tags, like <input>, which are self-closing).
  • Place attributes and contents, escaping special characters (but not the contents of the <style> and <script> tags).

Believe it or not, these are the patterns you need to create an unfolding dsDSL layer that generates HTML. Similar patterns can be found for generating CSS. In fact, lith does both, in ~250 lines of code.

One last question remains to be answered: What do I mean by walk, then slide? When we deal with a problem domain, we want to use a tool that delivers us from the nasty details of that domain. In other words, we want to sweep the low level under the rug, the faster the better. The walk, then slide approach proposes exactly the opposite: spend some time on the low level. Embrace its quirks, and understand which are essential and which can be avoided in the face of a set of real, varied, and useful problems.

After walking in the low level for some time and solving useful problems, you will have a sufficiently deep understanding of their domain. The patterns of representation and generation will then arise naturally; they are wholly derived from the nature of the problem they intend to solve. You can then write code that employs them. If they work, you will be able to slide through problems where you recently had to walk through them. Sliding means many things; it implies speed, precision and lack of friction. Maybe more importantly, this quality can be felt; when solving problems with this tool, do you feel like you’re walking through the problem, or do you feel that you’re sliding through it?

Maybe the most important thing about an unfolded tool is not the fact that it frees us from having to deal with the low level. Rather, by capturing the empiric patterns of repetition in the low level, a good high level tool allows us to understand fully the domain of applicability.

An unfolded tool will not just solve a problem - it will enlighten you about the problem’s structure.

So, don’t run away from a worthy problem. First walk around it, then slide through it.

About the author

Federico Pereiro, Netherlands
member since December 14, 2014
Federico is a full stack JavaScript developer and open source contributor. He works remotely with international customers, specifying, implementing, and provisioning web applications. His focus is in developing radically simple systems for solving complex problems. [click to continue...]
Hiring? Meet the Top 10 Freelance Software Developers for Hire in December 2016

Comments

Jacek Podkanski
Could we learn a lesson or two from Haskell?
fuzzylollipop
I know you make a big disclaimer about this being a rant and all. Your first straw man example is not valid, not all templating systems suffer from what you complain about. You just picked crappy ones. StringTemplate for example has no way to embed scripts into it. It is actually completely declarative, generating HTML and HTML fragments from it is extremely robust and easy. So 50% of your argument is not valid and your second argument is not valid for the same root cause reason, poor implementation of a tool is not a reflection on the declarative programming paradigm. It is 100% reflection on the implementation.
Federico Pereiro
Hi there! I would be very interested to read about similar developments in Haskell - or any language, for that matter. I know that Haskell is a functional language and its powerful pattern matching capabilities would leave a lot of room for this kind of approach. Did you think of anything in particular? Thanks for reading & commenting!
Antony Guinard
For the web, elm (http://elm-lang.org/) is an interesting functional/declarative language inspired by Haskell but with a steep learning curve! This link gives some insight about its architecture https://gist.github.com/evancz/2b2ba366cae1887fe621
Jacek Podkanski
might be the missing link https://en.wikipedia.org/wiki/Monad_%28functional_programming%29 http://moonbase.rydia.net/mental/writings/programming/monads-in-ruby/00introduction.html http://codon.com/refactoring-ruby-with-monads
Federico Pereiro
Hi there! l just checked on StringTemplate and it looks interesting and well-thought out. Reading the Philosophy section of it (https://github.com/antlr/stringtemplate4/blob/master/doc/motivation.md) I find the following statement: "After examining hundreds of template files that I created over years of jGuru.com (and now in ANTLR v3) development, I found that I needed only the following four basic canonical operations (with some variations): - attribute reference; e.g., - template reference (like #include or macro expansion); e.g., - conditional include of subtemplate (an IF statement); e.g., <title> - template application to list of attributes; e.g., names:bold() where template references can be recursive." My personal take on this is that I'd rather have more language features than the above four when I'm writing templates - the above set is powerful, but still I'd rather have the full power of the language available. Again, this is a strict matter of personal taste. Thanks for reading and commenting!
TimInColorado
I kept expecting you to mention React, Vue.js or Mithril: https://facebook.github.io/react/ http://vuejs.org/ https://lhorie.github.io/mithril/ ... since all have you write 100% JavaScript mixed with data or imperative declarations within JavaScript. I've also been struggling with the "separation of concerns" idea. I get why you've come down on the side of "it's still logic you need!", but it seems that all of the above approaches handicap one of the great advances of HTML: The ability for tools to allow designers to edit the layout without touching code. Because you're now putting most of your HTML in your JavaScript files, there's very little way for a tool to go in and edit anything. Does anyone use designers any more? Or is all the work just done by front-end developers these days?
Federico Pereiro
Hi Antony! It looks interesting, I hadn't heard of it until you mentioned it. From what I understood, it is a small, purely functional and powerful language that compiles to html/css/js. I cherish the fact that it uses code to generate views, albeit in a structured way. Thanks for sharing!
Federico Pereiro
Hi Tim! You raise many interesting points. Mithrill: I used it a few months ago and became very excited by the fact that it uses plain JS to create views! What I didn't like about it is that you need to invoke the `m` function for every tag, for example: ``` return m("html", [ m("body", [ m("input"), m("button", "Add") ]]); ``` I'd much rather write: `['html', ['body', [['input'], ['button', 'Add']]]]`. In more general terms, I would like to use expressions all the time, instead of a mixture of expressions and statements - and I feel that function applications written like this (the statements) cut the flow that you otherwise might achieve using 100% expressions. Regarding React, the fact that it still uses string interpolation and other DSL patterns makes me think I'll experience with it the same problems I had with other template libraries. I am not certain of this though - and of course it doesn't detract from all the other wonderful things that react does, in particular the use of a virtual DOM. The point about views becoming code is very accurate and I don't have a good response to that. This approach requires that someone should code the views - either from scratch, or using an HTML/CSS mockup as reference.
David Lormor
It feels like you've reduced declarative programming down to "using DSLs", which IMHO is probably the most complex implementation of a declarative programming style. To really "get" what declarative programming is all about I HIGHLY recommend Sandy Metz's Practical Object Oriented Design in Ruby. It's become pretty much required reading for any Ruby developer, and it really gets at the hard of what solid, OO-style declarative code is all about. It's all about expressing intent via messages passed between the objects in your system. I think the reason most templating language implementations become a mess is because lazy, overworked, undisciplined, or uninformed programmers don't get how (or take the time) to form well constructed objects that their system can interact with. Basically it comes down to good abstractions, well thought out methods, and good naming (the hardest thing in programming according to many a season veteran...). Anyways, that's enough ranting in response to your rant :) happy coding!
Federico Pereiro
Thank you for the links. It is good to read again about monads - I think this time I understood them ; ). It's very interesting to see how monads can represent computation and help avoid a lot of boilerplate. I think this provides the opportunity to not use a DSL and still avoid the repetition patterns of normal code.
fuzzylollipop
then you do not want to use declareitve templates or "programming" you want imperative, which is kind of the point of my comment. Your "article" is slamming one thing for not being the opposite of what it is. The things you claim are declarative are not. And the template systems that allow for imperative programming inside them in a DSL are not declarative, you entire treatise is based on incorrect assumptions, definitions and fallacy arguments.
fuzzylollipop
I think you mean "shallow learning curve" as a "steep" one means extremely easy to learn quickly.
Federico Pereiro
Hi David! It is entirely true that in this article I reduced declarative programming to DSLs. The critique of declarative programming I did above would not apply to other types of declarative programming, like functional programming (which I use constantly and actually make an indirect case for in the second section) and OO, constraint programming, etc. (of which I'm painfully ignorant). This was an over generalization of which I'm now aware - I stand corrected. My main bone to pick with the declarative tools I've come across (all of them DSL-based) is that they justified many awkward limitations by stating that it was better to lack that kind of power in a declarative layer. However it was mistaken to consider all these tools to represent the universe of declarative tools, as you rightly point out. I still think that DSL-based have systemic issues, which trascend mere "bad design". Your recommendation seems highly interesting, I hope to check it out soon. Thanks for the rant!!
Federico Pereiro
What I would like is a declarative layer that allows me to use more powerful constructs when I need them, but only when I need them. Imperative is definitely not the way to go and I think we're better off looking for tools that provide us with a high level layer on top of the imperative one. And I still think that DSLs are fundamentally limited. The approach I propose (which lays no claim to be the only one) is to use functional programming and literals to create a DSL within the language, instead of recreating a language within the DSL. As David points in another comment, similar approaches are employed in other kinds of declarative programming.
TimInColorado
Your complaint about Mithril seems subjective. Not sure how expressions harm anything; also, Mithril's "m" call just results in a set of JSON code. You can actually precompile Mithril to eliminate the calls: http://mithril.js.org/optimizing-performance.html Or you can use JSX to create Mithiril: https://github.com/insin/msx React doesn't really use string interpolation as much as escapes to get you back to pure JavaScript, as I understand things. Vue.js was a brain fart. Not appropriate. Sorry.
David Lormor
I think it's also important to recognize that declarative programming isn't something one simply gets from a tool. Declarative programming is very much a "style" and many tools (templating tools as an example) simply enforce a constraint around not providing you the ability to write imperative code. I find that, in templating languages, this serves to keep logic separated from template - you can argue the merits of whether this is good or bad - I personally have been around one too many WordPress projects gone awry, so I very much appreciate this constraint. It "guarantees" that my templates won't be code that makes the intent of the template unclear. I do agree that DSL-based systems present a set of challenges that should be approached with caution, but in general I find writing in a declarative style makes it easy to visit code weeks, months, or even years later and understand what the code _should_ be doing (it doesn't always mean that it is, but at least I know what the author of the code was hoping to achieve...if they named things well). Moral of the story: PROGRAMMING IS HARD! lol
Antony Guinard
Point taken, I never realized this expression was so ambiguous! I was referring to the popular meaning of "steep learning curve", i.e. "difficult to learn initially" as opposed to its original more technical meaning as discussed here: http://english.stackexchange.com/questions/6209/what-is-meant-by-steep-learning-curve.
ashenwolf
IMO, the whole dsDSL idea is actually a subset of a much more powerfull LISP's "code as data" concept and most of the Clojure's (the JVM based LISP dialect) HTML libraries I know (Hiccup and Enlive) are actually working in the manner you describe and it is indeed handy and convenient :)
Lê Anh Quân
Hi fuzzylollipop, I just checked the StringTemplate, and I think you and I have the different understanding of Declarative Programming. Declarative is not about opting out "script", or limiting its user from using powerful but dangerous tools. Declarative, in my opinion, is to help user achieve their business with the most readable/maintainable code, in that sense, if user has to put their script there, instead of forcing them to put the script (could be very short) somewhere else, and has to navigate files or lines to read the logic, the let them do (put the script inline with template)
fuzzylollipop
Declarative Programming is when you "declare" what should be and the language makes it happen. Imperative is when you tell it exactly what to do in what order. For example. Puppet is declarative, Chef is imperative. The first you "declare" what state you want the system to be in and Puppet makes it happen. Chef you have to write all the instructions explicitly to install and configure everything yourself. Your definition is wrong, plain and simple, if you have to tell the program what to do it is not declarative. Declarative programming : is a programming paradigm that expresses the logic of a computation(What do) without describing its control flow(How do). https://www.netguru.co/blog/imperative-vs-declarative
Lê Anh Quân
I you recommend you design and use your own DSL for the complicated business logic that you face, that's the best way to embrace Declarative Programming. A tailored DSL will contain only the details that your business concerned about, which lead to much simpler (suctinct) syntax for the DSL, and yet, provide maximum readability, maintainability. For example, I have a side bar here <<image 1>>, which is quite complicated. It contains not only the name and value of the field, but also "is that field required", "when to display that field" and "when that field is required". Here is what I came up with: { title: "DOB" , required: true, value: "form.patient_info.dob", filter: "date:'MM/dd/yyyy'"}, { title: "Gender" , required: true, value: "form.patient_info.gender"}, { title: "Phone 1", required: true, value: "form.patient_info.pri_phone"}, { title: "Phone 2", value: "form.patient_info.second_phone", condition: "form.patient_info.second_phone" }, ...
Federico Pereiro
Hi Ashenwolf! You hit the nail in the head. dsDSL is an extremely lispy idea. In any lisplike language, since programs themselves are expressed as lists, everything is code and data at the same time. The idea behind using as many expressions (and as few statements) as possible is precisely to make JS converge towards some sort of lisp, albeit with quite some more syntax. Hiccup and Enlive are quite close to the dsDSL approach I present above, especially Hiccup: ```hiccup (defn hello [] [:div {:class "well"} [:h1 {:class "text-info"} "Hello there"] ]) ``` ```lith function hello () { return lith.g (['div', {class: 'well'}, ['h1', {class: 'text-info'}, 'Hello there']]); } ``` dsDSLs are all about bringing lisplike qualities to JS - the fact that the language has first class functions, lambdas and data structures (arrays & objects) enable this possibility.
Federico Pereiro
Hi again, The problem I have with statements is that they are not first class. That is, they cannot be passed or returned from functions. In the case of Mithrill, every tag must be surrounded by an invocation to the "m" method, whereas in lith (a dsDSL library for generating HTML) you just write the tags as arrays and just do an invocation to the lith.g method in the outermost function. While you can surely write any HTML with both tools, it is considerably easier to just return arrays instead of returning function invocation of arrays, particularly when they can be arbitrarily nested. React uses JSX, which is "a JavaScript syntax extension that looks similar to XML". As such, I think it belongs to the class of DSLs, and hence the arguments in the first part of the article are relevant. And, like Mithrill, it profusely employs statements (instead of being almost entirely composed of expressions).
Federico Pereiro
Hi Lê! I wholeheartedly agree with your comment. Making your own DSL with JS arrays and objects can allow you to abstract the common patterns, especially in apps with tons of presentation and business logic. In the case of value, I see that you use the dot (".") as a separator. An approach that worked well for me is to use arrays of strings to represent paths. For your example, this would be: { title: "DOB" , required: true, value: ["form", "patient_info", "dob"], filter: "date:'MM/dd/yyyy'"}, { title: "Gender" , required: true, value: ["form", "patient_info", "gender"]}, { title: "Phone 1", required: true, value: ["form", "patient_info", "pri_phone"]}, { title: "Phone 2", value: "form.patient_info.second_phone", condition: ["form", "patient_info", "second_phone"]}, ... This allows you to have field names that contain dots or arbitrary characters. Also it allows you to do higher level operations on paths. I am exploring the possibilities of this approach for reactive programming and I hope to share it in the future. Thanks for reading & commenting!
David Lormor
I think you summed it up great: "I think that the essence of declarative programming is simply to provide a higher level layer that abstracts the repetition patterns of its low level." I call it writing code for humans - yes it requires a little more thought and effort to go beyond "it works," but I think we both agree that those benefits are well worth the cost! Now if only we could get our managers and clients to understand that working slow in this regard actually equals more productivity in the long term :)
Federico Pereiro
I couldn't agree more. And it is hard to convince managers and clients about slowness. The best solution I've come up so far is to work on my own "slowly made" tools, make them open source and then use them in customers' projects.
Federico Pereiro
Reading you I am also reminded of this quote by C. A. R. Hoare: "There is nothing a mere scientist can say that will stand against the flood of a hundred million dollars. But there is one quality that cannot be purchased in this way — and that is reliability. The price of reliability is the pursuit of the utmost simplicity."
Federico Pereiro
I'll start with the moral of the story: programming is hard indeed!! And there's no subsitute for effort and experience. Tools and approaches can only get you so far. However, it is good to build on patterns that can stand the test of time, and that's why we join the ongoing conversation about programming styles and approaches. I think that the essence of declarative programming is simply to provide a higher level layer that abstracts the repetition patterns of its low level. Having multiple levels make sense, as evidenced by the usefulness of assembler (a layer on top of object code), C (a layer on top of assembler) and dynamic languages ( (usually a layer on top of assembler or C). And I love declarative code, in the sense of code that frees you from repetition and obscureness. My idea here was to present a declarative approach that sidesteps traditional DSLs.
Leonardo Pullock
Developers in the Philippines your help is needed! I have this research project for Filipino Developers and them receiving payments. I have a possible reward for completing it https://form.jotform.com/61045669009154
TimInColorado
Mithril and React DOM construction methods both result in first class objects that can be passed into or returned from functions. Those objects can be combined into larger structures just as in your example. But both are more powerful than your dsDSL. And Mithril is crazy fast. JSX is not like a DSL, at least not in the sense of the article. Any logic is still plain JavaScript, so you don't suffer the disadvantages you sited in the article.
Lê Anh Quân
Hi Federico, About the "." problem, I agree that if you need dot in the field name, then we can not put it in the expression, but it's not the case with my convention (same as Java or Dot net rule - this is compulsory as I know). Also, "form.patient_info.dob" is more readable (less verbose) than ["form", "patient_info", "dob"]. So it all comes to the most important aspect of designing a DSL: how to make it most concise, short, readable, yet cover all the cases that our business require. In my case, "." would never happen. Still, we can assume having "." in the field name, and it can happen sometimes (not always). So it comes to another important practice which is "compromises": allowing exceptional, verbose cases to come to use. For example, if we need to have "." in the field name, we will make it like you propose, but keep other as my way, thus we will have: { title: "DOB" , required: true, value: "form.patient_info.dob", filter: "date:'MM/dd/yyyy'"}, { title: "Gender" , required: true, value: "form.patient_info.gender"}, { title: "Phone 1", required: true, value: ["form", "patient_info", "pri.phone"]}, { title: "Phone 2", value: "form.patient_info.second_phone", condition: "form.patient_info.second_phone" }, Note that the "pri.phone" field can now have ".", yet most of the times we don't need the verbose format you proposed. So with this mixed usage, now we have most fields easy to read, only 1 field looks hard to read, and ALL business cases are covered. Sorry for my bad/ugly English, but I guess you can understand what I'm trying to say
Lê Anh Quân
Hi fuzzylollipop, I understand your motive and your definition is correct, I wholeheartly agree. The problem with your thinking is you regard "script" as "imperative". Well, it's not. What about a part of "you declare what should be" is in "script"? Please take a look at this: student.height() > 1.6 This "script" will force the execution thread to invoke ".height()" method of the "student" object, then compare with 1.6... this makes we think that it's a "How do". Putting such a "script" in and code block would make you think that the code is "imperative". How about this: <div for-each="student in school" that-is="student.height() > 1.6">{{student.name}}</div> This is my "make up" template, and I think it's very much "declarative", how do you think?
Mauricio Souto da Rocha
Hi, nice article. I'd just recommend you to take a look on Lua. A programming language widely used for DSL development, specially on game mod development and configuration management. I'm not saying it's better or worse, nowadays I'd probably move towards using JavaScript and JSON for DSL/dsDSL development simply because everybody understands it and it has become a strong industry standard. Anyway, Lua is pretty interesting and worth using sometimes.
Federico Pereiro
Hi Mauricio! I played quite a while with Lua for Redis scripting purposes. I found the language to be powerful and pleasant to use - coming from Javascript I could adapt to it with minimal difficulty. I also think that the dsDSL approach is fully possible in Lua, thanks to its literal structures. Because Javascript is on every browser (and because I'm in a committed relationship with node.js) I don't plan on using another language for now - however, it's good to know that there are good alternatives around. Thanks for reading & commenting!
Federico Pereiro
Hi again Lê! I understand what you say and it's entirely tree that using arrays of strings is more verbose (for both reading and writing) than just using a string with a dot as a separator. The "path array" approach makes more sense if these attributes lists are generated automatically and then read again in an standard way - but that's another discussion. The important thing is to tailor your DSL to the precise requirements of the task at hand. I'm glad you're using this approach - thanks for sharing!
Federico Pereiro
Hi again, About Mithrill and React, you're right that the invocations return first class objects. However, I still consider the extra function invocations to be innecessary, especially in the case of Mithrill - just my personal take. Mithrill is crazy fast. The dsDSL I present above probably cannot compete with Mithrill in raw speed, because of the amount of runtime type detection that's necessary to enable it. In practice, I still found it to perform quite fast, even on mobile devices. You might be right about JSX - I have used it only very cursorily. Thanks for the clarification!
comments powered by Disqus
Subscribe
The #1 Blog for Engineers
Get the latest content first.
No spam. Just great engineering and design posts.
The #1 Blog for Engineers
Get the latest content first.
Thank you for subscribing!
You can edit your subscription preferences here.
Trending articles
Relevant technologies
About the author
Federico Pereiro
CSS Developer
Federico is a full stack JavaScript developer and open source contributor. He works remotely with international customers, specifying, implementing, and provisioning web applications. His focus is in developing radically simple systems for solving complex problems.