Chapter 8
Migrations
Why do we need Migrations
Once you get into production you'll need to synchronize model changes into the database. Typically, it is unsafe to use synchronize: true for schema synchronization on production once you get data in your database. Here is where migrations come to help.
A migration is just a single file with SQL queries to update a database schema and apply new changes to an existing database.
Step 1: Move TypeORM config into a separate file
You have to create a new folder with the db name in the root directory and create a new file data-source.ts
//db/data-source.ts
import { DataSource, DataSourceOptions } from "typeorm";
export const dataSourceOptions: DataSourceOptions = {
type: "postgres",
host: "localhost",
port: 5432,
username: "postgres",
password: "root",
database: "n-test",
entities: ["dist/**/*.entity.js"], //1
synchronize: false, // 2
migrations: ["dist/db/migrations/*.js"], // 3
};
const dataSource = new DataSource(dataSourceOptions); //4
export default dataSource;
Now you don't need to register the entity manually. TypeORM will find the entities by itself.
When you are working with migrations you have to set the synchronize to false because our migration file will update the changes in the database
You have to provide the path of migration where you want to store. I chose the dist folder. I will run the migrations as a js file. That's why we need to build the project before running typeorm migrations
We will use this data source object when we generate/run the migrations with typeorm cli
Step 2: Refactor TypeORM config in AppModule
//app.module.ts
import { dataSourceOptions } from "db/data-source";
imports: [TypeOrmModule.forRoot(dataSourceOptions)];
Step 3: Migration scripts in package.json file
//package.json
"typeorm": "npm run build && npx typeorm -d dist/db/data-source.js",
"migration:generate": "npm run typeorm -- migration:generate",
"migration:run" : "npm run typeorm -- migration:run",
"migration:revert" : "npm run typeorm -- migration:revert"
Step 4: Add a new column in any Entity
I am thinking about adding a phone column in the user entity. Maybe, the requirements changed after 3 months and you have to add a new column in the user entity.
//user.entity.ts
@Column()
phone: string
Now you have to update the user table using migrations.
npm run migration:generate -- db/migrations/add-user-phone
db/migrations : I am telling typeorm I want to save migrations in this folder
add-user-phone : This is the name of the migration
It will generate a new migration file inside the migrations folder
Now you have to run the migration by using this command
npm run migration:run
This command will alter/update the user table in the database
You can see the users table in the database and see the phone column there.
What is Data Seeding
Data seeding is the process of populating a database with an initial set of data. Applying seed data to a database refers to the process of inserting initial data into a database, usually when the database is first created. This data serves as a baseline and can be used for testing, and development, and to provide some context for the application that will be built on top of the database.
1. Install Dependencies
I am going to use an external package to generate fake/mock data.
npm install @faker-js/faker
2. Create a seed-data.ts file in the db/seeds directory
import { Artist } from "src/artists/artist.entity";
import { User } from "src/users/user.entity";
import { EntityManager } from "typeorm";
import { faker } from "@faker-js/faker";
import { v4 as uuid4 } from "uuid";
import * as bcrypt from "bcryptjs";
import { Playlist } from "src/playlist/playlist.entity";
export const seedData = async (manager: EntityManager): Promise<void> => {
//1
// Add your seeding logic here using the manager
// For example:
await seedUser();
await seedArtist();
await seedPlayLists();
async function seedUser() {
//2
const salt = await bcrypt.genSalt();
const encryptedPassword = await bcrypt.hash("123456", salt);
const user = new User();
user.firstName = faker.person.firstName();
user.lastName = faker.person.lastName();
user.email = faker.internet.email();
user.password = encryptedPassword;
user.apiKey = uuid4();
await manager.getRepository(User).save(user);
}
async function seedArtist() {
const salt = await bcrypt.genSalt();
const encryptedPassword = await bcrypt.hash("123456", salt);
const user = new User();
user.firstName = faker.person.firstName();
user.lastName = faker.person.lastName();
user.email = faker.internet.email();
user.password = encryptedPassword;
user.apiKey = uuid4();
const artist = new Artist();
artist.user = user;
await manager.getRepository(User).save(user);
await manager.getRepository(Artist).save(artist);
}
async function seedPlayLists() {
const salt = await bcrypt.genSalt();
const encryptedPassword = await bcrypt.hash("123456", salt);
const user = new User();
user.firstName = faker.person.firstName();
user.lastName = faker.person.lastName();
user.email = faker.internet.email();
user.password = encryptedPassword;
user.apiKey = uuid4();
const playList = new Playlist();
playList.name = faker.music.genre();
playList.user = user;
await manager.getRepository(User).save(user);
await manager.getRepository(Playlist).save(playList);
}
};
I have created a seedData method with an entity manager argument. I will get the repository for each entity by calling the getRepository. This is how you can create mock data
I have used the Faker package to generate mock data. You can see the functions faker.person.firstName(). You can explore methods from Faker by looking at the documentation
3. Create a new seed module
nest g module seed
import { Module } from "@nestjs/common";
import { SeedService } from "./seed.service";
@Module({
providers: [SeedService],
})
export class SeedModule {}
Make sure you have imported the SeedModule in AppModule
4.Create a SeedService
import { Injectable } from "@nestjs/common";
import { DataSource } from "typeorm";
import { seedData } from "../../db/seeds/seed-data";
@Injectable()
export class SeedService {
constructor(private readonly connection: DataSource) {}
async seed(): Promise<void> {
const queryRunner = this.connection.createQueryRunner(); //1
await queryRunner.connect(); //2
await queryRunner.startTransaction(); //3
try {
const manager = queryRunner.manager;
await seedData(manager);
await queryRunner.commitTransaction(); //4
} catch (err) {
console.log("Error during database seeding:", err);
await queryRunner.rollbackTransaction(); // 5
} finally {
await queryRunner.release(); //6
}
}
}
A Query Runner can be used to manage and work with a single real database data source. Each new QueryRunner instance takes a single connection from the connection pool if RDBMS supports connection pooling. For databases not supporting connection pools, it uses the same connection across data source.
Use the connect method to actually obtain a connection from the connection pool.
QueryRunner provides a single database connection. Transactions are organized using query runners. Single transactions can only be established on a single query runner. You can manually create a query runner instance and use it to manually control transaction state.
Commit the Transaction
If we have errors let's rollback changes we made
Make sure to release it when it is not needed anymore to make it available to the connection pool again
5. Run the Seeds
// main.ts
import { NestFactory } from "@nestjs/core";
import { AppModule } from "./app.module";
import { ValidationPipe } from "@nestjs/common";
import { SeedService } from "./seed/seed.service";
async function bootstrap() {
const app = await NestFactory.create(AppModule);
app.useGlobalPipes(new ValidationPipe());
const seedService = app.get(SeedService);
await seedService.seed();
await app.listen(3001);
}
bootstrap();
This is an entry file, Nestjs will run this file to bootstrap the application, whenever you run the application it will create new data and save it to DB. When you don't need new data you can disable these two lines:
// const seedService = app.get(SeedService);
// await seedService.seed();
Chapter 9 Configuration
Applications often run in different environments. Depending on the environment, different configuration settings should be used. For example, usually, the local environment relies on specific database credentials, valid only for the local DB instance. The production environment would use a separate set of DB credentials. Since configuration variables change, best practice is to store configuration variables in the environment.
In Nest.js, configurations refer to the settings and parameters that define the behavior of your application. These configurations can include various aspects, such as server settings, database connections, API keys, logging options, and more.
Nest.js provides a flexible way to manage configurations, allowing you to easily customize the behavior of your application based on different environments (e.g., development, production, testing) or specific deployment scenarios. By separating configuration settings from your code, you can make your application more portable and adaptable to different environments.
Configurations in Nest.js typically consist of key-value pairs, where each key represents a specific setting and the corresponding value represents its configuration value. These configurations are often stored in environment variables or configuration files.
By utilizing configurations, you can ensure that your application can be easily configured without modifying the code itself. This makes it simpler to deploy and maintain your application in different environments or when working with multiple teams.
Nest.js provides various mechanisms to load and use configurations, including the popular dotenv package for loading environment variables from files, as well as custom configuration modules and services that encapsulate the configuration logic and provide access to the configuration values throughout your application.
Overall, configurations in Nest.js help you manage the behavior of your application in a flexible and modular way, enabling you to adapt it to different environments and deployment scenarios without modifying the underlying code.
Step 1: Install Dependencies
"@nestjs/config": "^2.3.2",
Nest.js provides built-in config package for configuration. We are going to use this package. The @nestjs/config package internally uses dotenv.
Step 2: Import ConfigModule in AppModule
import { ConfigModule } from '@nestjs/config';
@Module({
imports: [
ConfigModule.forRoot(),
})
You have to provide the path of your env file in the ConfigModule.forRoot() method. We will have two env files, one is for development and the other is for production level
Step 3: Creating Custom Env Files
You have to .development.env and .production.env files in the root directory
.development.env
PORT = 3000;
.production.env
PORT = 3000;
You also have to register your env files in AppModule.
ConfigModule.forRoot({ envFilePath: ['.development.env', '.production.env'] }),
Step 4: Make ConfigModule to global
ConfigModule.forRoot({
isGlobal: true,
});
When you want to use ConfigModule in other modules, you'll need to import it (as is standard with any Nest module). Alternatively, declare it as a global module by setting the options object's isGlobal property to true, as shown below. In that case, you will not need to import ConfigModule in other modules once it's been loaded in the root module (e.g., AppModule)
Step 5: Creating CustomConfiguration.ts file
You have to create a new config folder inside the src directory and create new configuration.ts file there. You can create multiple custom configuration files for the database, app settings, and jwt, etc. A custom configuration file exports a factory function that returns a configuration object.
//config/configuration.ts
export default () => ({
port: parseInt(process.env.PORT),
});
Step 6: Load Configuration File
ConfigModule.forRoot({
envFilePath: ['.development.env', '.production.env'],
isGlobal: true,
load: [configuration],
}),
We load this file using the load property of the options object we pass to the ConfigModule.forRoot() method. The value assigned to the load property is an array, allowing you to load multiple configuration files (e.g. load: [databaseConfig, authConfig]
Step 7: Test the env variable
//auth.service.ts
import { ConfigService } from "@nestjs/config";
export class AuthService {
constructor(private configService: ConfigService) {}
getEnvVariables() {
return {
port: this.configService.get<number>("port"),
};
}
}
You can inject ConfigService as a dependency in any service. I wanted to show you the usage of ConfigService in a service. To access configuration values from our ConfigService, we first need to inject ConfigService. As with any provider, we need to import it's containing module - the ConfigModule - into the module that will use it (unless you set the isGlobal property in the options object passed to the ConfigModule.forRoot() method to true).
We don't need to import the ConfigModule in AuthModule because we have made it global in the AppModule
You can get the environment variables by using the get function from ConfigService. I have provided the type <number> and the key of the variable
Let's create the test method in the AuthController to get the env variable. I have created this route for only testing purposes
//auth.controller.ts
@Get('test')
testEnv() {
return this.authService.getEnvVariables();
}
### TEST ENV VARIABLES
GET http://localhost:3001/auth/test
When you send a request to this URL you will get the port number in the response
Step 8: Use ConfigService in main.ts file
When you open the main.ts file you will see the manual port value which is 3000. Let's get the value from ConfigService and add the port to the listen method. One more thing you can get the ConfigService instance from the app by calling app. get(ConfigService)
//main.ts
import { ConfigService } from "@nestjs/config";
async function bootstrap() {
const app = await NestFactory.create(AppModule);
const configService = app.get(ConfigService); // get the instance of ConfigService using app.get
await app.listen(configService.get<number>("port"));
}
bootstrap();
Step 9: Add JWT secret in Configuration
.development.env
SECRET=HAD_12X#@
.production.env
SECRET=HAD_12X#@
You also have to add the new value in the configuration.ts file for the secret.
//configuration.ts
secret: process.env.SECRET,
Now we need to add the jwt secret inside the AuthModule while registering the JwtModule
//auth.module.ts
import { ConfigModule, ConfigService } from '@nestjs/config';
JwtModule.registerAsync({
imports: [ConfigModule],
useFactory: async (configService: ConfigService) => ({
secret: configService.get<string>('secret'),
signOptions: {
expiresIn: '1d',
},
}),
inject: [ConfigService],
}),
The registerAsync method will return the DynamicModule. In Nest.js, a dynamic module is a feature that allows you to dynamically configure and register modules at runtime based on dynamic conditions or external factors. It provides a way to encapsulate complex configuration logic and allows modules to be created and registered programmatically.
Dynamic modules are useful when you have modules that require some dynamic configuration or when you want to conditionally load modules based on runtime conditions or variables.
static registerAsync(options: JwtModuleAsyncOptions): DynamicModule;
Step 10: Setup DB Configuration
.development.env
# Database Configuration for Development
DB_HOST=localhost
DB_PORT=5432
USERNAME=postgres
PASSWORD=root
DB_NAME=spotify-clone
We have used database configuration manually, now we need to get all the configuration from the env file. We can have a separate db configuration for the development and production environment. You have to use your database configuration like dbName, username, and password
You have to refactor the data-source.ts file
//data-source.ts
import { ConfigModule, ConfigService } from "@nestjs/config";
import {
TypeOrmModuleAsyncOptions,
TypeOrmModuleOptions,
} from "@nestjs/typeorm";
export const typeOrmAsyncConfig: TypeOrmModuleAsyncOptions = {
imports: [ConfigModule],
inject: [ConfigService],
useFactory: async (
configService: ConfigService
): Promise<TypeOrmModuleOptions> => {
return {
type: "postgres",
host: configService.get<string>("dbHost"),
port: configService.get<number>("dbPort"),
username: configService.get<string>("username"),
database: configService.get<string>("dbName"),
password: configService.get<string>("password"),
entities: ["dist/**/*.entity.js"],
synchronize: false,
migrations: ["dist/db/migrations/*.js"],
};
},
};
export const dataSourceOptions: DataSourceOptions = {
type: "postgres",
host: process.env.DB_HOST,
port: parseInt(process.env.DB_PORT),
username: process.env.USERNAME,
database: process.env.DB_NAME,
password: process.env.DB_PASSWORD,
entities: ["dist/**/*.entity.js"],
synchronize: false,
migrations: ["dist/db/migrations/*.js"],
};
I have created a new object typeOrmAsyncConfig inside the data-source.ts
Let's refactor the TyepOrm Module registration inAppModule
//app.module.ts
TypeOrmModule.forRootAsync(typeOrmAsyncConfig),
You have to add more properties in the configuration.ts file
//configuration.ts
dbHost: process.env.DB_HOST,
dbPort: parseInt(process.env.DB_PORT),
username: process.env.USERNAME,
password: process.env.PASSWORD,
dbName: process.env.DB_NAME,
Application Configurations
Add validation for environment variables by utilizing the class-validator package in your Nest.js application. This ensures that the application doesn't start with incorrect or missing configurations, adhering to the "Fail-fast" principle in software engineering. Throw the validation error before the application starts to avoid runtime issues.
Step 1: Add Node_ENV variable in .env files
Implement the validation for env variables. I am going to add the new property NODE_ENV=development in the .env.development file and NODE_ENV=production. We have to validate these variables
# .env.development
NODE_ENV=development
# .env.production
NODE_ENV=production
Step 2: Create .env.validation.ts
You have to add validation logic in this file. I have created this file inside the root folder.
import { plainToInstance } from "class-transformer";
import { IsEnum, IsNumber, IsString, validateSync } from "class-validator";
enum Environment {
Development = "development",
Production = "production",
Test = "test",
Provision = "provision",
}
class EnvironmentVariables {
// 1
@IsEnum(Environment)
NODE_ENV: Environment;
@IsNumber()
PORT: number;
@IsString()
DB_HOST: string;
@IsString()
USERNAME: string;
@IsString()
PASSWORD: string;
@IsString()
DB_NAME: string;
@IsString()
SECRET: string;
}
export function validate(config: Record<string, unknown>) {
//plainInstance converts plain (literal) object to class (constructor) object. Also works with arrays.
const validatedConfig = plainToInstance(EnvironmentVariables, config, {
/**
* enableImplicitConversion will tell class-transformer that if it sees a primitive that is currently a string (like a boolean or a number) to assume it should be the primitive type instead and transform it, even though @Type(() => Number) or @Type(() => Boolean) isn't used
*/
enableImplicitConversion: true,
});
/**
* Performs sync validation of the given object.
* Note that this method completely ignores async validations.
* If you want to properly perform validation you need to call validate method instead.
*/
const errors = validateSync(validatedConfig, {
skipMissingProperties: false,
});
if (errors.length > 0) {
throw new Error(errors.toString());
}
return validatedConfig;
}
We have added the validations for all our env variables. Whenever you need to create a new env variable in .env.development or .env.production file. You can create a validation rule inside the EnvironmentVariables class
Class-transformer allows you to transform a plain object to some instance of class and versa. Also, it allows the serialization/deserializing of objects based on criteria. This tool is super useful on both the front end and backend.
You can read the documentation of class-transformer here
Step 3: Add validate method in ConfigModule
You can add validate method in ConfigModule in AppModule
//app.module.ts
import { validate } from "env.validation";
ConfigModule.forRoot({
validate: validate,
});
I have noticed that when you make changes in your application it will take too much time to reload the application we can make this process faster by using webpack HMR
Step 1: Create webpack-hmr.config.js
First of all, you have to install this dev dependency
npm install -D run-script-webpack-plugin
You have to create this file in the root directory
// eslint-disable-next-line @typescript-eslint/no-var-requires
const nodeExternals = require("webpack-node-externals");
// eslint-disable-next-line @typescript-eslint/no-var-requires
const { RunScriptWebpackPlugin } = require("run-script-webpack-plugin");
module.exports = function (options, webpack) {
return {
...options,
entry: ["webpack/hot/poll?100", options.entry],
externals: [
nodeExternals({
allowlist: ["webpack/hot/poll?100"],
}),
],
plugins: [
...options.plugins,
new webpack.HotModuleReplacementPlugin(),
new webpack.WatchIgnorePlugin({
paths: [/\.js$/, /\.d\.ts$/],
}),
new RunScriptWebpackPlugin({
name: options.output.filename,
autoRestart: false,
}),
],
};
};
Step 2: Refactor bootstrap function in main.ts
declare const module: any;
async function bootstrap() {
////....
if (module.hot) {
module.hot.accept();
module.hot.dispose(() => app.close());
}
}
Step 3: Refactor start:dev script in package.json
"start:dev": "nest build --webpack --webpackPath webpack-hmr.config.js --watch",
You have to specificy that you want to reload the application using webpack. Now you test the application by making some changes and your application reload time would be faster
$7 bundle of my best NestJS + backend developer Courses.
More details coming soon.
Get the latest insights from the marketing world.
