0%

Integration/end-to-end testing is considered one of the best indicators that your code functions correctly, and everything is wired up as it should. Testcontainers is a wonderful library for creating and running embeddable Docker containers that can run alongside your tests, so you can have real implementations of your third-party dependencies, instead of relying on fragile mocks. Testcontainers is a mature Java library that comes out of the box with integrations for Postgres, Kafka, RabbitMQ, and many more, as well as support for many test runners and even ports to other languages.

In this short post, I’ll explain how to integrate Testcontainers (the Scala flavor) to play nicely with ZIO Test, showcasing some of the great composability features ZIO provides.

I will use a Postgres Testcontainer and Flyway to perform migrations, intializing the databse schemas.

Getting started

To play nicely with ZIO Test, we need to expose the Testcontainer as a ZLayer, so it can be added to ZIO’s Test Environment.

We’ll need to add the following dependencies to our build.sbt:

"org.flywaydb"   % "flyway-core"                      % flywayVersion,
"ch.qos.logback" % "logback-classic" % logbackVersion,
"com.dimafeng" %% "testcontainers-scala-postgresql" % testContainersVersion % Test

This adds Flyway to perform the actual migrations, Logback to print useful output messages from the container, and the Scala wrapper of the Postgres Test container. Replace the *Version values with the actual latest versions available for these dependencies.

Next, let’s add introduce Testcontainers to ZIO:

import com.dimafeng.testcontainers.PostgreSQLContainer
import zio.blocking.{effectBlocking, Blocking}
import zio._

object TestContainer {
type Postgres = Has[PostgreSQLContainer]

def postgres(imageName: Option[String] = Some("postgres")): ZLayer[Blocking, Nothing, Postgres] =
ZManaged.make {
effectBlocking {
val container = new PostgreSQLContainer(
dockerImageNameOverride = imageName,
)
container.start()
container
}.orDie
}(container => effectBlocking(container.stop()).orDie).toLayer
}

By default, the Scala Postgres container wrapper will fetch version 9 of PostgreSQL, you can get a specific version by specifying the imageName parameter with a specific tag, e.g. postgres:12.3, or leaving postgres to fetch the latest.

We use ZIO’s ZManaged to wrap the creation and disposal of the container, wrapping the actual creation and shutdown in effectBlocking to signal to ZIO that this should be done on the blocking thread pool. This makes our ZLayer require the Blocking service. On shutdown, we stop the container. If creating or stopping fails for any reason, we’d like our test to terminate, which we do with .orDie, making any potential failures to be treated as defects, causing ZIO to shut down. Finally, we turn the ZManaged into a ZLayer by calling toLayer on it.

And that’s it! We now have a Postgres layer to plug into the ZIO tests.

Performing migrations

Once our container has started, we want to populate the database with our schema, before the tests are run. For this, ZIO Tests provides a mechanism called Test Aspects, which allows, among other things, to execute an action before executing the tests. We can create a Test Aspect that runs the migration:

import com.dimafeng.testcontainers.PostgreSQLContainer
import org.flywaydb.core.Flyway
import zio.ZIO
import zio.blocking.effectBlocking
import zio.test.TestAspect.before

object MigrationAspects {
def migrate(schema: String, paths: String*) = {
val migration = for {
pg <- ZIO.service[PostgreSQLContainer]
_ <- runMigration(pg.jdbcUrl, pg.username, pg.password, schema, paths: _*)
} yield ()

before(migration.orDie)
}

private def runMigration(
url: String,
username: String,
password: String,
schema: String,
locations: String*
) =
effectBlocking {
Flyway
.configure()
.dataSource(url, username, password)
.schemas(schema)
.locations(locations: _*)
.load()
.migrate()
}
}

Here we define a function migrate that takes in the name of the schema for the migration, as well as the paths where the migration scripts are located. Because the container assigns a random port each time it starts, we fetch the container service and perform the migration using its parameters (such as the jdbcUrl). Again, because Flyway is a Java API, we wrap it with effectBlocking to ensure ZIO runs it on the blocking thread pool.

Putting it all together

Finally, we can put it all together into a test that looks like this:

object MyPostgresIntegrationSpec extends DefaultRunnableSpec {
val postgresLayer = Blocking.live >>> TestContainer.postgres()
val testEnv = zio.test.environment.testEnvironment ++ postgresLayer

val spec = suite("Postgres integration") {
testM("Can create and fetch a customer") {
for {
userId <- CustomerService.create("testUser", "testPassword")
result <- CustomerService.find(userId)
} yield assert(result.id)(equalTo(userId)) &&
assert(result.username)(equalTo("testUser"))
}
}.provideCustomLayer(testEnv) @@ migrate("customers", "filesystem:src/customers/resources/db/migration")

We create a test environment testEnv by combining the default ZIO Test environment layer with the Postgres layer, and passing it to the test suite using provideCustomLayer. In addition, we perform the migration at the start of each test by applying migrate using the @@ operator. Finally, depending on the organization of your modules, you can specify custom paths to your migration scripts, as well as the schema name.

And that’s it, we now have an integration test that starts Postgres, performs the migration, and runs your test. When it’s finished, ZIO gracefully shuts down the container!

Bonus: making it nicer

Here are a few tweaks you can do to the above setup:

Specifying a default schema

By default, the Postgres Testcontainer does not provide a way to specify a currentSchema parameter on the JDBC URL. We can fix this with some good, old-fashioned inheritance:

final class SchemaAwarePostgresContainer(
dockerImageNameOverride: Option[String] = None,
databaseName: Option[String] = None,
pgUsername: Option[String] = None,
pgPassword: Option[String] = None,
mountPostgresDataToTmpfs: Boolean = false,
currentSchema: Option[String] = None
) extends PostgreSQLContainer(
dockerImageNameOverride,
databaseName,
pgUsername,
pgPassword,
mountPostgresDataToTmpfs
) {
override def jdbcUrl: String =
currentSchema.fold(super.jdbcUrl)(schema => s"${super.jdbcUrl}&currentSchema=$schema")
}

We’ll extend PostgreSQLContainer to allow specifying the currentSchema as a constructor argument (in addition to all the others), and will override jdbcUrl to append it to the URL (if was specified). Now, let’s replace our ZLayer creation function with the new class:

type Postgres = Has[SchemaAwarePostgresContainer]

def postgres(currentSchema: Option[String] = None): ZLayer[Blocking, Nothing, Postgres] =
ZManaged.make {
effectBlocking {
val container = new SchemaAwarePostgresContainer(
dockerImageNameOverride = Some("postgres"),
currentSchema = currentSchema
)
container.start()
DockerLoggerFactory.getLogger(container.container.getDockerImageName).info(s"⚡ ${container.jdbcUrl}")
container
}.orDie
}(container => effectBlocking(container.stop()).orDie).toLayer

Note that the parameter to postgres() now accepts the schema name instead of the docker image name, the container will fetch the latest available PostgresSQL image. Tweak this according to your needs.

In addition, I’ve added an log output line using the Testcontainer’s Docker logger to print out the JDBC URL for easier debugging.

Creating an ITSpec for easier integration tests

Last thing we can do is to create a base class to hide all this work, allowing us to write integration tests without adding any of the boilerplate. This was adapted from an excellent library called tranzactio by Gaël Renoux, which helps Doobie (and Anorm) play nice with ZIO:

object ITSpec {
type ITEnv = TestEnvironment with Logging with Postgres
}

abstract class ITSpec(schema: Option[String] = None) extends RunnableSpec[ITEnv, Any] {
type ITSpec = ZSpec[ITEnv, Any]

override def aspects: List[TestAspect[Nothing, ITEnv, Nothing, Any]] =
List(TestAspect.timeout(60.seconds))

override def runner: TestRunner[ITEnv, Any] =
TestRunner(TestExecutor.default(itLayer))

val itLayer: ULayer[ITEnv] = {
val postgres = Blocking.live >>> TestContainer.postgres(schema)
val logging = Slf4jLogger.make(LogFmtRenderer())
... // other services you might want to create for testing
testEnvironment ++ logging ++ postgres
}.orDie
}

Which allows us to create Postgres-powered integration specs like this:

object MyITSpec extends ITSpec(Some("schemaName")) {
val spec: ITSpec = suite("Integration suite")(
testM("...") { }
) @@ migrate("schemaName", "filesystem:path/to/resources/for/migration")
}

And the ITSpec base suite will provide all the necessery environments for us! Note that val spec: ITSpec must have ITSpec type ascription, otherwise Scala will not be able to correctly infer all the requirements.

Happy testing!

A few days ago I tweeted a C# code snippet, showing a FizzBuzz implementation using some of the new features in C# 8.0. The tweet “went viral”, as the kids say, with several people admiring the terse and functional aspect of it, while others asked me why I wasn’t writing it in F# in the first place?

Read more »

Over the past two years, the PDF version of Bartosz Milewski’s Category Theory for Programmers became a highly-successful open-source book, which was adapted to other programming languages, such as Scala and OCaml. Unfortunately, building this 400+ page PDF from LaTeX sources in multiple editions took a significant amount of time, sometimes upwards of 15 minutes. One of the reasons was, all the code snippets are loaded from external code files (so that they can be easily adapted to other programming languages), and they have to be compiled each time in a format LaTeX understands.

I recently explored some possibilities to reduce the time it takes for the snippets to build, and I’m happy to report the results: a 60% improvement overall! Big thanks to muzimuzhi from the minted github, who generously helped me to arrive at the solution below.

Here’s a quick summary of my changes, that resulted in reducing my Travis CI builds from 14-16 minutes to mere 6!

Read more »

This was initially a long post, detailing all the manual steps required to set up a complete Haskell development environment, however, thanks to a hint by Krzysztof Cieślak, this process is now fully automated, allowing you to get started in minutes. All thanks to a Visual Studio Code feature called devcontainers, supporting running the development environment in a Docker container.

Read more »

There’s a fantastic free online course (MOOC) for the Russian-speaking developer community on Stepik for learning Haskell - a two-part course titled Functional Programming in Haskell by Denis Moskvin, (then) associate professor at the St. Petersburg Academic University. I recently re-watched the course (having completed it previously) and decided to take notes and summarize the course content in English for your enjoyment.

I would like to thank Denis Moskvin for providing this amazing resource for free, and urge you, if you speak Russian and want to learn Haskell, to work through the course material and exercises!

Below is the summary of the first module, Introduction, out of 5.

Read more »

It’s amazing how sometimes just having a different framing of the problem helps with developing a much deeper understanding of the problem. I was working through the exercises of the Data61 Functional Programming course, assisted by Brian McKenna’s video streams, and I came accross a definition of a right fold that can be thought of as “constructor replacement”:

The expression foldr f z list replaces in list:

  1. Every occurence of the cons constructor (:) with f
  2. Any occurrence of the nil constructor [] with z
Read more »

The book The Pragmatic Programmer: From Journeyman to Master by Andy Hunt and Dave Thomas suggests that as developers, we should “learn at least one new language every year.” (pg. 14)

When I recently asked a roomful of developers, if there’s anyone who had learned a new language this year, only very few hands went up. A year ago today, that would have been me in the audience, keeping my hands down.

Read more »

Original title was “Monads solve a problem you might not have, but it’s a nice problem to have”, which is an homage to a great post by Krzysztof Koźmic about IoC containers.

I can’t think of another 5-letter word that strikes fear in the hearts of so many developers, coming from an object-oriended/imperative language to a functional one. So much so, this, and other M-words are outright banned on some resources.

This post will not attempt to explain monads, at least, not on purpose. This fantastic post by Max Kreminski does this better than I ever could - by showing that most “monad tutorials” (or, educational blog posts in general) have problem-solution ordering issues. Please take a moment to read this wonderful post before continuing.

Read more »