Elixir Umbrella Projects: Building Blocks for Code that Scales

By Pedro Assumpcao, Lead Software Engineer

In my last CityBase blog post on Elixir, I discussed how Elixir helps make some of the most complex systems work more effectively.

One of the features of Elixir that helps accomplish that goal is the ability to work with umbrella projects.

What is an umbrella app?

From the Erlang documentation: “In Open Telecom Protocol (OTP), application denotes a component implementing some specific functionality, that can be started and stopped as a unit, and that can be reused in other systems.”

For a regular Elixir project, you have only one application, responsible for all the logic of your system, following the concept from the documentation.

With umbrella projects, Elixir allows you to have more than one application in your system. Each additional application has to conform to the concept of an application already explained. Applications inside an umbrella can have siblings as dependencies as well.

The really nice thing about this is that there is almost no difference between an application inside an umbrella and the same application if it were built as a single project. Also, this similarity applies to external dependencies as well. In reality, all of this follows the same concept of an OTP application.

Benefits of umbrella projects

One of the positive points in an umbrella project is that it helps you think about your system with clear separation of logic and responsibilities, which translates to the organization of your code. This brings to the table clear cognitive benefits for team members who are joining the project, and for future maintenance.

The definition of components is very clear: the interdependencies between applications and how they relate to each other are transparent and easy to find.

Another interesting situation is when an application is identified as a good candidate for an external library. To extract one application and make it a library is really straightforward, as libraries are based on the same concept of application.

Scalability

Because applications inside an umbrella are stand-alone components, and based on how Elixir projects are released, it is possible to have different strategies for deployment. In practice, it is common for umbrella projects to be deployed as a single release. However, Elixir offers the option of having releases per application or a group of applications.

So, in a scenario where one application inside an umbrella has higher demand (of any kind), for scalability purposes it makes sense to deploy the higher demand application to more nodes than all the other applications.

Umbrella cons

Another point to highlight is the fact that common external libraries must be added as dependencies and set up in all applications if you want to have them available internally. Common libraries are usually for testing and development, such as Credo, Dialyxir, ExDoc, and ExCoveralls. In other Elixir projects, all applications live in the main mix.exs file.

Monolith vs Microservices vs Umbrella

Instead, consider umbrella as a middle ground. It brings a lot of the benefits of both approaches, and also minimizes some of the negative points in each.

The following list (and I am not considering possible anti-patterns) shows some of the costs of both approaches, monolith or microservice, and how umbrella projects minimize those:

Monolith costs:

2) Libraries used for a small part of the application are considered required for the entire system.

3) Understanding a monolith requires extra effort, which affects the development of new features.

4) Any change, small or big, requires a full system deployment.

5) Debugging can become a complex task as contracts between parts are not clear.

Microservice costs:

7) Communication between services will require one more layer of complexity (transport and message protocol) and fault-tolerance.

8) Tracing errors across services is complex as errors in one specific service are local.

Umbrella benefits that minimize costs above:

2) Based on the same principle above, libraries that belong to one application are not tied to other applications.

3) The idea of having completely separate applications by design helps understand which pieces of the system are important, and it also exposes the contracts between applications.

4) It is possible to deploy individual applications within an umbrella, enhancing scalability over time as well.

5) Logic separation of the applications, and explicit contracts between them, will isolate errors and simplify debugging.

6) Applications inside an umbrella are part of a single project so everything is available automatically.

7) Communication between applications that are deployed as separate nodes is made through Erlang Port Mapper Daemon. Messages are Erlang terms and there is no need for an additional serialization protocol.

8) Tracing is available out-of-box, and it works exactly as if you have a single Elixir project.

An example of an umbrella project evolution

Later, inside the umbrella, new applications, with specific responsibilities can be added. For example, a web application to be an interface for core:

And then, we realize that users will have to use an authentication mechanism to get access to some functions. Since authentication is a separate concern from core and web, and authentication can rely on third party libraries, it makes more sense to build its logic as a separate application inside the umbrella:

The idea here is that new scenarios can happen as the project evolves, and all of them can leverage the umbrella design:

1) We may identify other needs that are not core business logic and those should have a proper home.

2) We may decide to open source our auth application, extract it to a hex package. Doing so is very simple as it is already working as a standalone application.

3) We may need other types of interfaces besides web, so each interface could be an application inside the umbrella using the same core (and maybe auth) functionality.

Why CityBase is in favor of umbrella projects

Additionally, reducing the need of communication logic between applications is really important when handling many services, as you don’t need to spawn one service per functionality or commingle functionalities in the same service, creating a monolith.

Originally published at CityBase.

We’re working with innovative public servants to transform the experience of government.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store