Integration testing using Testcontainers in a Spring Boot 2, JPA application

In this post we will be looking at how to perform integration testing of the repository layer using Testcontainer. We will be setting up Testcontainer in a Spring Boot 2 application, use JPA for persistence and PostgreSQL as database.Besides this we will also be using JUnit5 framework.

Why Integration testing of the database ?

In any software application, it is a known fact that data is of utmost importance. In a web application, this data will usually be handled at the database layer. In an application which uses JPA for persistence, the repository and the JPA entities are major components of the database layer.

To ensure that the repository layer and the entities are functioning appropriately, we should write unit tests. Unit tests for the same are usually written against an in memory database. This is definitely required. The only catch here – Should we wait until production to ensure that the database layer is behaving as expected against our actual type of our database as against the in memory database ?

What would be even better is if we can do integration testing against a similar database that we use in production environment – MySQL,Oracle,PostgreSQL etc. But wouldn’t setting this up be difficult and laborious for just integration testing ?

Enter Testcontainers

Using Testcontainers, we can actually start a containerized instance of the database to test our database layer without going through the pain of actually setting it up on every developer’s machine. Testcontainer is a Java library that provides disposable and lightweight instances of databases. It actually does a lot more that just that. You can read about it here.

How does it help us ?

Using JUnit tests against the actual type of our production database will give us feedback before we actually go into production.This means we can worry less about our database layer and breathe a little easy. There will always be other issues in production,NullPointerExceptions in our code that we can worry about! Since Testcontainers spawns a containerized environment for us to run our tests, we need to have Docker installed.

Getting started with the code

We will be using Spring Boot 2.3.0 along with Spring Data JPA, JUnit 5 to run the tests,Flyway to create tables and TestContainer version 1.14.1.

Snippet of the pom.xml
<!-- The PostgreSQL db driver -->        
<dependency>
	<groupId>org.postgresql</groupId>
	<artifactId>postgresql</artifactId>
	<scope>test</scope>
</dependency> 
<dependency>
	<groupId>org.testcontainers</groupId>
	<artifactId>testcontainers</artifactId>
	<version>${testcontainer.version}</version>
	<scope>test</scope>	
</dependency>
<!-- test container support for postgre module -->
<dependency>
        <groupId>org.testcontainers</groupId>
        <artifactId>postgresql</artifactId>
        <version>${testcontainer.version}</version>
        <scope>test</scope>
</dependency>
<!-- JUnit 5 integration with testcontainer -->
<dependency>
	<groupId>org.testcontainers</groupId>
	<artifactId>junit-jupiter</artifactId>
	<version>${testcontainer.version}</version>
	<scope>test</scope>
</dependency>
The application.properties file
spring.jpa.hibernate.ddl-auto=none
spring.jpa.database-platform=org.hibernate.dialect.PostgreSQL94Dialect
spring.jpa.show-sql=true

The spring.jpa.hibernate.ddl-auto has been set to none because we are using flyway to create our database and tables. The initialization script can be found under the /db/migration folder.

Starting the containerized PostgreSQL database and tying it with Spring
public abstract class AbstractContainerBaseTest {

	protected static final PostgreSQLContainer postgres;

	static {
		postgres = new PostgreSQLContainer("postgres:9.6.15")
										 .withDatabaseName("socialmediasite")
										 .withUsername("postgres")
										 .withPassword("password");
		//Mapped port can only be obtained after container is started.
		postgres.start();
	}
	
	static class PropertiesInitializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {
		public void initialize(ConfigurableApplicationContext configurableApplicationContext) {			
			TestPropertyValues
					.of("spring.datasource.driver-class-name=" + postgres.getDriverClassName(),
							"spring.datasource.url=" + postgres.getJdbcUrl(),
							"spring.datasource.username=" + postgres.getUsername(),
							"spring.datasource.password=" + postgres.getPassword())
					.applyTo(configurableApplicationContext.getEnvironment());
		}
	}
}

The main purpose of this abstract class is to start the container using Testcontainer library. All other test classes will extend this class .The static block creates the container and this instance will be shared by all test classes. There is a manual call to start method. The library manages to shut it down automatically.

The PropertiesInitializer class is needed because we need to access this containerized database through the Spring Data JPA.

Initialization of Testcontainer,PostgreSQL, Spring Boot,Spring Data JPA
Verifying the output
PostgreSQL container created and closed by Testcontainer

I am using the docker ps –format command to list the containers started by Testcontainer library. As shown in the image above, one can see the container having the name competent_raman which is the PostgreSQL container.For all the test cases this is the single instance of the container. There is also this additional image quay.io/testcontainers/ryuk:0.2.3 – this is created by the Testcontainer library which ensures all other containers are shut down and cleaned when the JVM shuts down.

Database and tables in the containerized PostgreSQL database.

As you can see in the image above, I am using the docker exec -it command to connect to the same containerized instance, (competent_raman) of the PostgreSQL database. Then using the command psql -h 192.168.99.100 -p 32813, we are connected to the dockerized database. The -p indicates the port number and -h the host. The \c socialmediasite command connects us to the database and finally the \d command is used to list all the tables.

Once all the tests are executed, the Testcontainer library removes the instances of the containers that it created.

The JPA entities SocialMediaSite and User used in the application have a One-Many relationship.

Conclusion

Writing unit tests against an in-memory database like H2 is certainly helpful but writing integration tests against actual type of production database takes us to the next level – more confidence, fewer failures in production. The combination of Testcontainers, Spring Boot 2, JUnit5 makes it easy for us to get started on this journey. You can find the entire source code here.

Understanding @SpringBootApplication

The objective of this post is to get an understanding of what @SpringBootApplication annotation does in a Spring Boot application. Sometimes we just tend to add an annotation, everything works magically and we are happy.

But when we want to make a few changes, things start breaking, components are not found and then we blame the magic that we enjoyed earlier.So it is always a good idea to get a better understanding of what some of these annotations do.

In a Spring Boot application, we need a class with a main method annotated with @SpringBootApplication annotation as shown below:

package com.boot.jpa;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class SpringBootJpaApplication {

	public static void main(String[] args) {
		SpringApplication.run(SpringBootJpaApplication.class, args);
	}

}

This is the starting point for a Spring Boot application. The @SpringBootApplication annotation is really a short hand annotation or a combination of the following annotations:

1. @SpringBootConfiguration

This signifies that the application is not just a normal Spring application but a Spring Boot application. Actually this is just a combination of @Configuration+ Spring Boot, so more of a specialization. @Configuration applied on a file signifies that the class contains spring bean configurations using @Bean.

2. @EnableAutoConfiguration

This is the super intelligent annotation which kicks in the auto configuration for Spring Boot. Auto configuration is done by inspecting the classpath and configuring necessary beans that may be required. If you were to provide your own configuration, then Spring Boot will not re-configure the bean again.

How is this auto configuration done ?

Auto Configuration is done via a SpringFactoriesLoader class which will pick up and read the META-INF/spring.factories file.  It is like a hook into instantiating certain beans found on the classpath. A sample of the same is shown below. EnableAutoConfiguration class is the ‘key’ below and it can have many ‘values’ as classes. These are read and conditionally configured depending on what is already configured by the developer and what is found on the classpath. For example, only if RabbitMQ and Spring AMQP client are found on the classpath usually added via maven or gradle, then Spring Boot tries to configure RabbitMQ.

# Auto Configure
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.boot.autoconfigure.amqp.RabbitAutoConfiguration,\
org.springframework.boot.autoconfigure.data.jpa.JpaRepositoriesAutoConfiguration,\
....

3. @ComponentScan

Any Spring application will usually consist of beans which will be managed by Spring framework. To tell Spring framework to manage a bean, we use @ComponentScan. This annotation will scan all your beans and register it with the Spring Application Context. Later, you could inject the beans using @Autowired. Examples of a Component are classes annotated with @Controller, @Repository, @Service etc.

One has to be careful as @ComponentScan will scan the packages below the package from where this is defined.

A better way to organize your packages would be to have the  class containing the @SpringBootApplication in the root package of your application so that all sub packages get scanned and beans are registered with application context. If this is not possible for some reason, you can of course add @ComponentScan annotation and specify the base package(s) to scan for spring beans.

Conclusion

Annotations in the Spring or Spring Boot framework makes the life of a developer easy and increases productivity.However it is important to understand the ‘what and how’ of these annotations to be a more effective and efficient developer.

Testing JPA entities using Spring Boot 2, JUnit 5 and Java 14

In this article we will be looking at how to get started with testing JPA entities and the repository layer using Spring Boot 2.2 , JUnit 5 and Java 14. I have written a similar post here which uses Spring 1.5, JUnit 4 and Java 8. As the changes are significant I decided to keep them separate instead of updating that post.

I will be focusing mostly about the changes I had to make to the code to upgrade the libraries mentioned above. You can find the complete source code on github here.

Let us consider the same example of One-Many relation between SocialMediaSite and a User.

The @DataJpaTest

This remains the key ingredient behind running the JPA related tests in Spring Boot. @DataJpaTest disables full auto configuration and applies configuration related to JPA tests only. This concept hasn’t changed between Spring Boot 1.5 and 2.x. Now, let’s take a look at the areas where the code needed modifications.

Changes to the pom.xml

Upgrading the spring boot version to 2.2.6

<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.2.6.RELEASE</version>
		<relativePath /> <!-- lookup parent from repository -->
</parent>

Upgrading java version to 14

<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
		<java.version>14</java.version>
	</properties>

Upgrading the dependency section to exclude JUnit 4

The spring-boot-starter-test dependency includes the vintage JUnit 4 and JUnit 5 dependencies.Since we will be using JUnit 5, we will exclude the vintage one.

<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
			<exclusions>
				<exclusion>
					<groupId>org.junit.vintage</groupId>
					<artifactId>junit-vintage-engine</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

Junit 5 consists of 3 different sub projects – The JUnit engine , JUnit vintage ( Junit 3 and 4) and JUnit Jupiter( JUnit5). All JUnit 5 annotations reside in org.junit.jupiter.api package.

Support for preview features in Java 14

<plugin>
	<artifactId>maven-compiler-plugin</artifactId>
	<configuration>
		<release>14</release>
		<compilerArgs>
		    <arg>--enable-preview</arg>
		 </compilerArgs>
					 
                <forceJavacCompilerUse>true</forceJavacCompilerUse>
		  <parameters>true</parameters>
	</configuration>
</plugin>

<plugin>
	<artifactId>maven-surefire-plugin</artifactId>
	<configuration>
		<argLine>--enable-preview</argLine>
	</configuration>
</plugin>
Use of the Record type

Record type has been added as a preview feature in Java 14. The EmailAddress class has now been modified as a Record type. This was a value object before.

public record EmailAddress(String emailAddress) {

	public static final Pattern VALID_EMAIL_ADDRESS_REGEX = Pattern.compile("^[A-Z0-9._%+-]+@[A-Z0-9.-]+\\.[A-Z]{2,6}$",
			Pattern.CASE_INSENSITIVE);

	public EmailAddress {
		Assert.isTrue(isValid(emailAddress), "Email Address is Invalid !");
	}

	private static boolean isValid(String emailAddress) {
		return emailAddress != null ? VALID_EMAIL_ADDRESS_REGEX.matcher(emailAddress).matches() : false;
	}

}
Changes to the Test classes
  1. There is no need to add the @RunWith(SpringRunner.class) annotation anymore.
  2. The @Before annotation has been now replaced with @BeforeEach to perform any kind of setup before each test. Similarly the @After has been replaced with @AfterEach.
  3. The @Rule and @ExpectedException had been removed from JUnit5. The test classes have been refactored to reflect the same. The assertThrows and assertEquals method from the JUnit5 library have been used instead.
@Test
public void testShouldReturnExceptionForInvalidEmailAddress() {
		
	var exception = assertThrows(IllegalArgumentException.class, () -> new EmailAddress("test@.com"));
	assertEquals(exception.getMessage(), "Email Address is Invalid !");
}

The assertThrows method takes a functional interface, Executable as second parameter. We pass the block of code that needs to be executed to Executable.

Conclusion

The @DataJpaTest helps us to test the JPA entities and the repository layer. This is also called as test slicing as we are testing only the JPA components.The @DataJpaTest annotation contains @ExtendWith(SpringExtension.class) which integrates the Spring Testing framework with JUnit 5 model.

We have not explored many features of JUnit5 in this example but the combination of Spring Boot’s testing features and JUnit5 does make a powerful combination.

Getting Intimate with Spring Boot and Hibernate

In my previous blog , we looked at how to get started with a simple Spring Boot and Hibernate application. We managed to get our application up and running in a few minutes.

In this article, we will look at how and what Spring Boot does behind the scenes.

Basic requirement

To setup a Spring based application with Hibernate/JPA we usually need the following:

  1. Datasource  which connects to a database and details like the database url, password, username – this is really independent of whether we use Spring, JPA.
  2. JPA EntityManager to perform repository(CRUD) related operations.
  3. Vendor specific (Hibernate) properties
  4. Transaction support.

Let us take a look at how these things get configured.

In case of an in memory H2 database, all of the above was configured automatically by Spring Boot. We did not write a single configuration either using Java configuration or XML. Well, how does Spring Boot do everything for us?  Let’s take a look :

Vanishing (configuration) act explained :

1. The starting point is the spring.factories file. This file has a Auto Configure section which Spring Boot uses to determine what should be auto configured. This file is in the META-INF folder which is part of the spring-boot-autoconfigure-<version>.RELEASE.jar. The SpringFactoriesLoader class loads the spring.factories file.

2. Since we are talking about JPA/Hibernate, the spring.factories contains a (among other autoconfigurations) key value pair: org.springframework.boot.autoconfigure.EnableAutoConfiguration=\org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration

3. The key EnableAutoConfiguration above is triggered using the @SpringBootApplication annotation on the SpringBootJpaApplication.java file. This is our starting point in our application.

package com.boot.jpa;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
@SpringBootApplication
public class SpringBootJpaApplication {
public static void main(String[] args) {
SpringApplication.run(SpringBootJpaApplication.class, args);
}
}

4. The value part above, HibernatJpaAutoConfiguration class which is part of the Spring framework looks like

@Configuration
@ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class })
@Conditional(HibernateEntityManagerCondition.class)
@AutoConfigureAfter({ DataSourceAutoConfiguration.class })
public class HibernateJpaAutoConfiguration extends JpaBaseConfiguration {
….
}

One of the important annotations above is on line 4 :

@AutoConfigureAfter({ DataSourceAutoConfiguration.class }).

This indicates that we should perform HibernateJpaAutoConfiguration only after reading the DataSourceAutoConfiguration.class or only if a Datasource is found on the classpath. It would make no sense to configure JPA and Hibernate without a datasource/database !

5. The DataSourceAutoConfiguration looks as follows:

@Configuration
@ConditionalOnClass({ DataSource.class, EmbeddedDatabaseType.class })
@EnableConfigurationProperties(DataSourceProperties.class)
@Import({ Registrar.class, DataSourcePoolMetadataProvidersConfiguration.class })
public class DataSourceAutoConfiguration {
}

6. This is a normal Spring Configuration class which sets up the DataSource only if all of the above annotations satisfy our criteria. We used an H2 database and hence the DataSource.class and EmbeddedDatabaseType.class satisfies the condition.

7. Now the configuration moves back to step 4 where we needed the presence of  datasource/database at minimum. Now on to the next part where the following is checked :

@ConditionalOnClass({ LocalContainerEntityManagerFactoryBean.class, EntityManager.class })

This is configuration specific to JPA classes and the @ConditionalOnClass checks if these classes are there on the class path. There are present in our case through the Spring Data JPA library.

8.Next comes the @Conditional(HibernateEntityManagerCondition.class) . This checks if the JPA provider is Hibernate, this is done via the HibernateEntityManagerCondition.java. In our case it is true since Spring Boot brings in Hibernate as the default provider of JPA and HibernateEntityManager.java is on the classpath.

9.Once these conditions are satisfied, the HibernateJpaAutoConfiguration extends the JpaBaseConfiguration class.

@EnableConfigurationProperties(JpaProperties.class)
@Import(DataSourceInitializedPublisher.Registrar.class)
public abstract class JpaBaseConfiguration implements BeanFactoryAware {
}

This is annotated with the @EnableConfigurationProperties(JpaProperties.class).  This JpaProperties.class is annotated with @ConfigurationProperties(prefix = “spring.jpa”) . This gets triggered if we were to add any spring.jpa.xxx properties in our application.properties file. We would use this if we were to add any jpa specific properties. We did not add any and hence the default is assumed.

10.The JpaBaseConfiguration class contains all the other remaining configurations. It primarily uses the @ConditionalOnMissingBean annotation to configure the EntityManager, JpaVendorAdapter, LocalContainerEntityManagerFactoryBean, JpaTransactionManager if they have already not been configured. The @ConditionalOnMissingBean when applied on a method means – if this bean has not been configured, then execute the method and configure it.

@Bean
@ConditionalOnMissingBean(PlatformTransactionManager.class)
public PlatformTransactionManager transactionManager() {
JpaTransactionManager transactionManager = new JpaTransactionManager();
return transactionManager;
}

So in this way, datasource, entitymanager, vendor(hibernate) specific properties and transactions get configured. If you have setup a Spring Hibernate application before, we would do the same using Java configuration or an xml and configure the above mentioned properties. You can configure any specific properties and in that case Spring Boot will skip auto configuring that property.

Conclusion 

All the Harry Houdini illusions or Spring Boot magic starts from the spring.factories file and then through the usage of some powerful annotations like @ConditionalOnMissingBean, ConditionalOnClass,@AutoConfigureAfter , Jpa/Datasource/Transactions gets configured depending on what is found in the classpath. Spring Boot will not configure a property or a bean if it was already configured.

Complaining is a Habit ?

I have heard people complaining about the so-called magic in Spring Boot. Well, when Spring folks gave us xml configurations, we complained ! They gave us configurations via annotations and Java Configuration to get rid of the xml, we still complained ! Now, things get configured automatically…..and we still complain. Complaining can be good at times, the folks at Pivotal have been listening to our complaints and have been giving us fantastic tools/ frameworks. But let’s not forget that the details are out there, let’s explore a little more before complaining.