Federico Pereiro
Maker of minimalist software. Designs, writes, tests, deploys and maintains real systems solving real (if boring) problems.
Expertise
Experience
12 years
In a nutshell, declarative programming consists of instructing a program on what needs to be done, instead of telling it how to do it. This approach involves providing a domain-specific language (DSL) for expressing what the user wants. This DSL shields users from messy low-level constructs while still achieving the desired end-state.
While declarative programming offers advantages over the imperative approach it replaces, it’s not as straightforward as it may seem. In this comprehensive article, Toptal Freelance Software Engineer Federico Pereiro outlines his experience with declarative tools and explains how you can make declarative programming work for you.
In a nutshell, declarative programming consists of instructing a program on what needs to be done, instead of telling it how to do it. This approach involves providing a domain-specific language (DSL) for expressing what the user wants. This DSL shields users from messy low-level constructs while still achieving the desired end-state.
While declarative programming offers advantages over the imperative approach it replaces, it’s not as straightforward as it may seem. In this comprehensive article, Toptal Freelance Software Engineer Federico Pereiro outlines his experience with declarative tools and explains how you can make declarative programming work for you.
Maker of minimalist software. Designs, writes, tests, deploys and maintains real systems solving real (if boring) problems.
12 years
Declarative programming is, currently, the dominant paradigm of an extensive and diverse set of domains such as databases, templating and configuration management.
In a nutshell, declarative programming consists of instructing a program on what needs to be done, instead of telling it how to do it. In practice, this approach entails providing a domain-specific language (DSL) for expressing what the user wants, and shielding them from the low-level constructs (loops, conditionals, assignments) that materialize the desired end state.
While this paradigm is a remarkable improvement over the imperative approach that it replaced, I contend that declarative programming has significant limitations, limitations that I explore in this article. Moreover, I propose a dual approach that captures the benefits of declarative programming while superseding its limitations.
CAVEAT: This article emerged as the result of a multi-year personal struggle with declarative tools. Many of the claims I present here are not thoroughly proven, and some are even presented at face value. A proper critique of declarative programming would take considerable time, effort, and I would have to go back and use many of these tools; my heart is not in such an undertaking. The purpose of this article is to share a few thoughts with you, pulling no punches, and showing what worked for me. If you’ve struggled with declarative programming tools, you might find respite and alternatives. And if you enjoy the paradigm and its tools, don’t take me too seriously.
If declarative programming works well for you, I’m in no position to tell you otherwise.
Before we explore the limits of declarative programming, it is necessary to understand its merits.
Arguably the most successful declarative programming tool is the relational database (RDB). It might even be the first declarative tool. In any case, RDBs exhibit the two properties that I consider archetypical of declarative programming:
Before RDBs, most database systems were accessed through imperative code, which is heavily dependent on low-level details such as the order of records, indexes and the physical paths to the data itself. Because these elements change over time, code often stops working because of some underlying change in the structure of the data. The resulting code is hard to write, hard to debug, hard to read and hard to maintain. I’ll go out a limb and say that most of this code was in, all likelihood, long, full of proverbial rats’ nests of conditionals, repetition and subtle, state-dependent bugs.
In the face of this, RDBs provided a tremendous productivity leap for systems developers. Now, instead of thousands of lines of imperative code, you had a clearly defined data scheme, plus hundreds (or even just tens) of queries. As a result, applications had only to deal with an abstract, meaningful and lasting representation of data, and interface it through a powerful, yet simple query language. The RDB probably raised the productivity of programmers, and companies that employed them, by an order of magnitude.
What are the commonly listed advantages of declarative programming?
While the above are all commonly cited advantages of declarative programming, I would like to condense them into two qualities, which will serve as guiding principles when I propose an alternative approach.
In the following two sections, I will present the two main problems of declarative programming: separateness and lack of unfolding. Every critique needs its bogeyman, so I will use HTML templating systems as a concrete example of the shortcomings of declarative programming.
Imagine that you need to write a web application with a non-trivial number of views. Hard coding these views into a set of HTML files is not an option because many components of these pages change.
The most straightforward solution, which is to generate HTML by concatenating strings, seems so horrible that you will quickly look for an alternative. The standard solution is to use a template system. Although there are different types of template systems, we will sidestep their differences for the purpose of this analysis. We can consider all of them to be similar in that the main mission of template systems is to provide an alternative to code that concatenates HTML strings using conditionals and loops, much like RDBs emerged as an alternative to code that looped through data records.
Let’s suppose we go with a standard templating system; you will encounter three sources of friction, which I will list in ascending order of importance. The first is that the template necessarily resides in a file separate from your code. Because the templating system uses a DSL, the syntax is different, so it cannot be in the same file. In simple projects, where file counts are low, the need to keep separate template files may duplicate or treble the amount of files.
I open an exception for Embedded Ruby templates (ERB), because those are integrated into Ruby source code. This is not the case for ERB-inspired tools written in other languages since those templates must also be stored as different files.
The second source of friction is that the DSL has its own syntax, one different from that of your programming language. Hence, modifying the DSL (let alone writing your own) is considerably harder. To go under the hood and change the tool, you need to learn about tokenizing and parsing, which is interesting and challenging, but hard. I happen to see this as a disadvantage.
You may ask, “Why on earth would you want to modify your tool? If you are doing a standard project, a well-written standard tool should fit the bill.” Maybe yes, maybe no.
A DSL never has the full power of a programming language. If it did, it wouldn’t be a DSL anymore, but rather a full programming language.
But isn’t that the whole point of a DSL? To not have the full power of a programming language available, so that we can achieve abstraction and eliminate most sources of bugs? Maybe, yes. However, most DSLs start simple and then gradually incorporate a growing number of the facilities of a programming language until, in fact, it becomes one. Template systems are a perfect example. Let’s see the standard features of template systems and how they correlate to programming language facilities:
This argument, that a DSL is limited because it simultaneously covets and rejects the power of a programming language, is directly proportional to the extent that the features of the DSL are directly mappable to the features of a programming language. In the case of SQL, the argument is weak because most of the things SQL offers are nothing like what you find in a normal programming language. At the other end of the spectrum, we find template systems where virtually every feature is making the DSL converge towards BASIC.
Let’s now step back and contemplate these three quintessential sources of friction, summed up by the concept of separateness. Because it is separate, a DSL needs to be located on a separate file; it is harder to modify (and even harder to write your own), and (often, but not always) needs you to add, one by one, the features you miss from a real programming language.
Separateness is an inherent problem of any DSL, no matter how well designed.
We now turn to a second problem of declarative tools, which is widespread but not inherent.
If I had written this article a few months ago, this section would have been named Most Declarative Tools Are #@!$#@! Complex But I Don’t Know Why. In the process of writing this article I found a better way of putting it: Most Declarative Tools Are Way More Complex Than They Need To Be. I will spend the rest of this section explaining why. To analyze the complexity of a tool, I propose a measure called the complexity gap. The complexity gap is the difference between solving a given problem with a tool versus solving it in the lower level (presumably, plain imperative code) that the tool intends to replace. When the former solution is more complex than the latter, we are in presence of the complexity gap. By more complex, I mean more lines of code, code that’s harder to read, harder to modify and harder to maintain, but not necessarily all of these at the same time.
Please note that we’re not comparing the lower level solution against the best possible tool, but rather against no tool. This echoes the medical principle of “First, do no harm”.
Signs of a tool with a large complexity gap are:
I might have fallen prey to emotion here since template systems are not that complex, but this comparatively small complexity gap is not a merit of their design, but rather because the domain of applicability is quite simple (remember, we’re just generating HTML here). Whenever the same approach is used for a more complex domain (such as configuration management) the complexity gap may quickly turn your project into a quagmire.
That said, it is not necessarily unacceptable for a tool to be somewhat more complex than the lower level it intends to replace; if the tool yields code that is more readable, concise and correct, it can be worth it t. It’s an issue when the tool is several times more complex than the problem it replaces; this is flat-out unacceptable. Brian Kernighan famously stated that, “Controlling complexity is the essence of computer programming.” If a tool adds significant complexity to your project, why even use it?
The question is, why are some declarative tools so much more complex than they need be? I think it would be a mistake to blame it on poor design. Such a general explanation, a blanket ad-hominem attack on the authors of these tools, is not fair. There has to be a more accurate and enlightening explanation.
My contention is that any tool that offers a high level interface to abstract a lower level must unfold this higher level from the lower one. The concept of unfolding comes from Christopher Alexander’s magnum opus, The Nature of Order - in particular Volume II. It is (hopelessly) beyond the scope of this article (not to mention my understanding) to summarize the implications of this monumental work for software design; I believe its impact will be huge in years to come. It is also beyond this article to provide a rigorous definition of unfolding processes. I will use here the concept in a heuristic way.
An unfolding process is one that, in a stepwise fashion, creates further structure without negating the existing one. At every step, each change (or differentiation, to use Alexander’s term) remains in harmony with any previous structure, when previous structure is, simply, a crystallized sequence of past changes.
Interestingly enough, Unix is a great example of the unfolding of a higher level from a lower one. In Unix, two complex features of the operative system, batch jobs and coroutines (pipes), are simply extensions of basic commands. Because of certain fundamental design decisions, such as making everything a stream of bytes, the shell being a userland program and standard I/O files, Unix is able to provide these sophisticated features with minimal complexity.
To underline why these are excellent examples of unfolding, I would like to quote a few excerpts of a 1979 paper by Dennis Ritchie, one of the authors of Unix:
On batch jobs:
… the new process control scheme instantly rendered some very valuable features trivial to implement; for example detached processes (with
&
) and recursive use of the shell as a command. Most systems have to supply some sort of specialbatch job submission
facility and a special command interpreter for files distinct from the one used interactively.
On coroutines:
The genius of the Unix pipeline is precisely that it is constructed from the very same commands used constantly in simplex fashion.
This elegance and simplicity, I argue, comes from an unfolding process. Batch jobs and coroutines are unfolded from previous structures (commands run in a userland shell). I believe that because of the minimalist philosophy and limited resources of the team that created Unix, the system evolved stepwise, and as such, was able to incorporate advanced features without turning its back on to the basic ones because there weren’t enough resources to do otherwise.
In the absence of an unfolding process, the high level will be considerably more complex than necessary. In other words, the complexity of most declarative tools stem from the fact that their high level does not unfold from the low level they intend to replace.
This lack of unfoldance, if you forgive the neologism, is routinely justified by the necessity to shield the user from the lower level. This emphasis on poka-yoke (protecting the user from low level errors) comes at the expense of a large complexity gap that is self-defeating because the extra complexity will generate new classes of errors. To add insult to injury, these classes of errors have nothing to do with the problem domain but rather with the tool itself. We would not go too far if we describe these errors as iatrogenic.
Declarative templating tools, at least when applied to the task of generating HTML views, are an archetypical case of a high level that turns its back on the low level it intends to replace. How so? Because generating any non-trivial view requires logic, and templating systems, especially logic-less ones, banish logic through the main door and then smuggle some of it back through the cat door.
Note: An even weaker justification for a large complexity gap is when a tool is marketed as magic, or something that just works, the opaqueness of the low level is supposed to be an asset because a magic tool is always supposed to work without you understanding why or how. In my experience, the more magical a tool purports to be, the faster it transmutes my enthusiasm into frustration.
But what about the separation of concerns? Shouldn’t view and logic remain separate? The core mistake, here, is to put business logic and presentation logic in the same bag. Business logic certainly has no place in a template, but presentation logic exists nevertheless. Excluding logic from templates pushes presentation logic into the server where it is awkwardly accommodated. I owe the clear formulation of this point to Alexei Boronine, who makes an excellent case for it in this article.
My feeling is that roughly two thirds of the work of a template resides in its presentation logic, while the other third deals with generic issues such as concatenating strings, closing tags, escaping special characters, and so on. This is the two-faced low level nature of generating HTML views. Templating systems deal appropriately with the second half, but they don’t fare well with the first. Logic-less templates flat out turn their back on this problem, forcing you to solve it awkwardly. Other template systems suffer because they truly need to provide a non-trivial programming language so their users can actually write presentation logic.
To sum up; declarative templating tools suffer because:
I would like to close the critique with an argument that is logically disconnected from the thread of this article, but deeply resonates with its emotional core: We have limited time to learn. Life is short, and on top of that, we need to work. In the face of our limitations, we need to spend our time learning things that will be useful and withstand time, even in the face of fast changing technology. That is why I exhort you to use tools that don’t just provide a solution but actually shed a bright light on the domain of its own applicability. RDBs teach you about data, and Unix teaches you about OS concepts, but with unsatisfactory tools that don’t unfold, I’ve always felt I was learning the intricacies of a sub-optimal solution while remaining in the dark about the nature of problem it intends to solve.
The heuristic I suggest you to consider is, value tools that illuminate their problem domain, instead of tools that obscure their problem domain behind purported features.
To overcome the two problems of declarative programming, which I have presented here, I propose a twin approach:
A data structure DSL (dsDSL) is a DSL that is built with the data structures of a programming language. The core idea is to use the basic data structures you have available, such as strings, numbers, arrays, objects and functions, and combine them to create abstractions to deal with a specific domain.
We want to keep the power of declaring structures or actions (high level) without having to specify the patterns that implement these constructs (low level). We want to overcome the separateness between the DSL and our programming language so that we are free to use the full power of a programming language whenever we need it. This is not only possible but straightforward through dsDSLs.
If you asked me a year ago, I would have thought that the concept of dsDSL was novel, then one day, I realized that JSON itself was a perfect example of this approach! A parsed JSON object consists of data structures that declaratively represent data entries in order to get the advantages of the DSL while also making it easy to parse and handle from within a programming language. (There might be other dsDSLs out there, but so far I haven’t come across any. If you know of one, I would really appreciate your mentioning it in the comments section.)
Like JSON, a dsDSL has the following attributes:
parse
and stringify
.But dsDSLs go beyond JSON in many ways. Let’s create a dsDSL for generating HTML using Javascript. Later I will touch on the issue of whether this approach may be extended to other languages (spoiler: It can definitely be done in Ruby and Python, but probably not in C).
HTML is a markup language composed of tags
delimited by angle brackets (<
and >
). These tags may have optional attributes and contents. Attributes are simply a list of key/value attributes, and contents may be either text or other tags. Both attributes and contents are optionals for any given tag. I’m simplifying somewhat, but it is accurate.
A straightforward way to represent an HTML tag in a dsDSL is by using an array with three elements:
- Tag: a string.
- Attributes: an object (of the plain, key/value type) or undefined
(if no attributes are necessary).
- Contents: a string (text), an array (another tag) or undefined
(if there’s no contents).
For example, <a href="views">Index</a>
can be written as ['a', {href: 'views'}, 'Index']
.
If we want to embed this anchor element into a div
with class links
, we can write: ['div', {class: 'links'}, ['a', {href: 'views'}, 'Index']]
.
To list several html tags at the same level, we can wrap them in an array:
[
['h1', 'Hello!'],
['a', {href: 'views'}, 'Index']
]
The same principle may be applied to creating multiple tags within a tag:
['body', [
['h1', 'Hello!'],
['a', {href: 'views'}, 'Index']
]]
Of course, this dsDSL won’t get us far if we don’t generate HTML from it. We need a generate
function which will take our dsDSL and yield a string with HTML. So if we run generate (['a', {href: 'views'}, 'Index'])
, we will get the string <a href="views">Index</a>
.
The idea behind any DSL is to specify a few constructs with a specific structure which is then passed to a function. In this case, the structure that makes up the dsDSL is this array, which has one to three elements; these arrays have a specific structure. If generate
thoroughly validates its input (and it is both easy and important to thoroughly validate input, since these validation rules are the precise analog of a DSL’s syntax), it will tell you exactly where you went wrong with your input. After a while, you’ll start to recognize what distinguishes a valid structure in a dsDSL, and this structure will be highly suggestive of the underlying thing it generates.
Now, what are the merits of a dsDSL in contraposition to a DSL?
Now, the last claim is a strong one, so I’m going to spend the rest of this section supporting it. What do I mean by properly employed? To see this in action, let’s consider an example in which we want to construct a table to display the information from an array named DATA
.
var DATA = [
{id: 1, description: 'Product 1', price: 20, onSale: true, categories: ['a']},
{id: 2, description: 'Product 2', price: 60, onSale: false, categories: ['b']},
{id: 3, description: 'Product 3', price: 120, onSale: false, categories: ['a', 'c']},
{id: 4, description: 'Product 4', price: 45, onSale: true, categories: ['a', 'b']}
]
In a real application, DATA
will be generated dynamically from a database query.
Moreover, we have a FILTER
variable which, when initialized, will be an array with the categories we want to display.
We want our table to:
id
field, but add it as an id
attribute for each row. ALTERNATE VERSION: Add an id
attribute to each tr
element.onSale
if the product is on sale.FILTER
is an empty array, we will display all products. Otherwise, we will only display the products where the category of the product is contained within FILTER
.We can create the presentation logic that matches this requirement in ~20 lines of code:
function drawTable (DATA, FILTER) {
var printableFields = ['description', 'price', 'categories'];
DATA.sort (function (a, b) {return a.price - b.price});
return ['table', [
['tr', dale.do (printableFields, function (field) {
return ['th', field];
})],
dale.do (DATA, function (product) {
var matches = (! FILTER || FILTER.length === 0) || dale.stop (product.categories, true, function (category) {
return FILTER.indexOf (category) !== -1;
});
return matches === false ? [] : ['tr', {
id: product.id,
class: product.onSale ? 'onsale' : undefined
}, dale.do (printableFields, function (field) {
return ['td', product [field]];
})];
})
]];
}
I concede this is not a straightforward example, however, it represents a fairly simple view of the four basic functions of persistent storage, also known as CRUD. Any non-trivial web application will have views that are more complex than this.
Let’s now see what this code is doing. First, it defines a function, drawTable
, to contain the presentation logic of drawing the product table. This function receives DATA
and FILTER
as parameters, so it can be used for different data sets and filters. drawTable
fulfills the double role of partial and helper.
var drawTable = function (DATA, FILTER) {
The inner variable, printableFields
, is the only place where you need to specify which fields are printable ones, avoiding repetition and inconsistencies in the face of changing requirements.
var printableFields = ['description', 'price', 'categories'];
We then sort DATA
according to the price of its products. Notice that different and more complex sort criteria would be straightforward to implement since we have the entire programming language at our disposal.
DATA.sort (function (a, b) {return a.price - b.price});
Here we return an object literal; an array which contains table
as its first element and its contents as the second. This is the dsDSL representation of the <table>
we want to create.
return ['table', [
We now create a row with the table headers. To create its contents, we use dale.do which is a function like Array.map, but one that also works for objects. We will iterate printableFields
and generate table headers for each of them:
['tr', dale.do (printableFields, function (field) {
return ['th', field];
})],
Notice that we have just implemented iteration, the workhorse of HTML generation, and we didn’t need any DSL constructs; we only needed a function to iterate a data structure and return dsDSLs. A similar native, or user-implemented function, would have done the trick as well.
Now iterate through the products contained in DATA
.
dale.do (DATA, function (product) {
We check whether this product is left out by FILTER
. If FILTER
is empty, we will print the product. If FILTER
is not empty, we will iterate through the categories of the product until we find one that is contained within FILTER
. We do this using dale.stop.
var matches = (! FILTER || FILTER.length === 0) || dale.stop (product.categories, true, function (category) {
return FILTER.indexOf (category) !== -1;
});
Notice the intricacy of the conditional; it is precisely tailored to our requirement and we have total freedom for expressing it because we are in a programming language rather than a DSL.
If matches
is false
, we return an empty array (so we don’t print this product). Otherwise, we return a <tr>
with its proper id and class and we iterate through printableFields
to, well, print the fields.
return matches === false ? [] : ['tr', {
id: product.id,
class: product.onSale ? 'onsale' : undefined
}, dale.do (printableFields, function (field) {
return ['td', product [field]];
Of course we close everything that we opened. Isn’t syntax fun?
})];
})
]];
}
Now, how do we incorporate this table into a wider context? We write a function named drawAll
that will invoke all functions that generate the views. Apart from drawTable
, we might also have drawHeader
, drawFooter
and other comparable functions, all of which will return dsDSLs.
var drawAll = function () {
return generate ([
drawHeader (),
drawTable (DATA, FILTER),
drawFooter ()
]);
}
If you don’t like how the above code looks, nothing I say will convince you. This is a dsDSL at its best. You might as well stop reading the article (and drop a mean comment too because you’ve earned the right to do so if you’ve made it this far!). But seriously, if the code above doesn’t strike you as elegant, nothing else in this article will.
For those who are still with me, I would like to go back to the main claim of this section, which is that a dsDSL has the advantages of both the high and the low level:
But how is this truly different from purely imperative code? I think ultimately the elegance of the dsDSL approach boils down to the fact that code written in this way mostly consists of expressions, instead of statements. More precisely, code that uses a dsDSL is almost entirely composed of:
Code that consists mostly of expressions and which encapsulate most statements within functions is extremely succinct because all patterns of repetition can be easily abstracted. You can write arbitrary code as long as that code returns a literal that conforms to a very specific, non-arbitrary form.
A further characteristic of dsDSLs (which we don’t have time to explore here) is the possibility of using types to increase the richness and succinctness of the literal structures. I will expound on this issue on a future article.
Might it be possible to create dsDSLs beyond Javascript, the One True Language? I think that it is, indeed, possible, as long as the language supports:
I think this means that dsDSLs are tenable in any modern dynamic language (i.e.: Ruby, Python, Perl, PHP), but probably not in C or Java.
In this section I will attempt to show a way for unfolding a high level tool from its domain. In a nutshell, the approach consists of the following steps
Now, what the hell are patterns of representation and patterns of generation? I’m glad you asked. The patterns of representation are the patterns in which you should be able to express a problem that belongs to the domain that concerns your tool. It is an alphabet of structures that allows you to write any pattern you might wish to express within its domain of applicability. In a DSL, these would be the production rules. Let’s go back to our dsDSL for generating HTML.
The patterns of representation for HTML are the following:
['TAG']
['TAG', {attribute1: value1, attribute2: value2, ...}]
['TAG', 'CONTENTS']
['TAG', {attribute1: value1, ...}, 'CONTENTS']
['TAG1', ['TAG2', ...]]
[['TAG1', ...], ['TAG2', ...]]
condition ? ['TAG', ...] : []
/ Depending on a condition, place an attribute or no attribute: ['TAG', {class: condition ? 'someClass': undefined}, ...]
These instances can be represented with the dsDSL notation we determined in the previous section. And this is all you need to represent any HTML you might need. More sophisticated patterns, such as conditional iteration through an object to generate a table, may be implemented with functions that return the patterns of representation above, and these patterns map directly to HTML tags.
If the patterns of representation are the structures you use to express what you want, the patterns of generation are the structures your tool will use to convert patterns of representation into the lower level structures. For HTML, these are the following:
<input>
, which are self-closing).<style>
and <script>
tags).Believe it or not, these are the patterns you need to create an unfolding dsDSL layer that generates HTML. Similar patterns can be found for generating CSS. In fact, lith does both, in ~250 lines of code.
One last question remains to be answered: What do I mean by walk, then slide? When we deal with a problem domain, we want to use a tool that delivers us from the nasty details of that domain. In other words, we want to sweep the low level under the rug, the faster the better. The walk, then slide approach proposes exactly the opposite: spend some time on the low level. Embrace its quirks, and understand which are essential and which can be avoided in the face of a set of real, varied, and useful problems.
After walking in the low level for some time and solving useful problems, you will have a sufficiently deep understanding of their domain. The patterns of representation and generation will then arise naturally; they are wholly derived from the nature of the problem they intend to solve. You can then write code that employs them. If they work, you will be able to slide through problems where you recently had to walk through them. Sliding means many things; it implies speed, precision and lack of friction. Maybe more importantly, this quality can be felt; when solving problems with this tool, do you feel like you’re walking through the problem, or do you feel that you’re sliding through it?
Maybe the most important thing about an unfolded tool is not the fact that it frees us from having to deal with the low level. Rather, by capturing the empiric patterns of repetition in the low level, a good high level tool allows us to understand fully the domain of applicability.
An unfolded tool will not just solve a problem - it will enlighten you about the problem’s structure.
So, don’t run away from a worthy problem. First walk around it, then slide through it.
Located in Leiden, Netherlands
Member since November 4, 2015
Maker of minimalist software. Designs, writes, tests, deploys and maintains real systems solving real (if boring) problems.
12 years
World-class articles, delivered weekly.
World-class articles, delivered weekly.
Join the Toptal® community.