Learn how to consume data from the MongoDB database in your Selenium WebDriver tests in a simple way using MongoDB Java Driver.
When developing a test automation script for your application you might need test data, or maybe your application has been deployed and the product owner or the business owner comes around the corner. The registration screen loads up but you realize there’s no data in the system…
Part 1: Getting Started with MongoDB
- What is MongoDB?
- MongoDB Features
- MongoDB Concepts (Databases and Collections)
- Install and Configure MongoDB Community Edition locally
- Install MongoDB Compass (GUI tool for MongoDB)
Part 2: Setup our Project
- The Prerequisites
- Create a new Gradle project
- Install and configure the project
- The CRUD operations using MongoDB Java Driver
Part 3: Integrate Selenium WebDriver with MongoDB Database
- Create Page Object class
- Create BaseTest Class
- Create the First test with Selenium WebDriver and MongoDB
Part 4: Continuous Integration with GitHub Actions
You’ll learn all this by implementing a simple web test script with Java and inside the script will allow you to create, update and get the data from MongoDB.
Let’s get started…
Part1: Getting Started with MongoDB
What is MongoDB?
MongoDB is a document-oriented NoSQL database management system written in the C ++ programming language. Since the database is document-oriented, it can manage collections of JSON-like documents.
- MongoDB stores data in flexible, JSON-like documents, meaning fields can vary from document to document and the data structure can change over time.
- The document model is mapped to the objects in your application code to simplify data processing
- Ad hoc queries, indexing, and real-time aggregation provide powerful ways to access and analyze your data.
- MongoDB is a distributed database at its core, so horizontal scaling and geographical distribution are integrated and easy to use.
The Difference between MongoDB and SQL databases
MongoDB offers both a Community and an Enterprise version of the database:
- MongoDB Community is the free-to-use edition of MongoDB.
- MongoDB Enterprise is available as part of the MongoDB Enterprise Advanced subscription and includes comprehensive support for your MongoDB deployment. MongoDB Enterprise also adds enterprise-focused features such as on-disk encryption, and auditing.
MongoDB Features
High availability through integrated replication and failover:
- Horizontal scalability with native sharding
- End-to-end security
- Native document validation and schema examination with Compass
- Management tools for automation, monitoring, and backup
- Fully elastic database as a service with built-in best practices
- Rich Query Language MongoDB query language is the way we communicate with MongoDB Data. From queries to updates, pipelines to batch jobs, everything keeps getting more powerful over time.
MongoDB Concepts (Collections and Documents)
MongoDB stores data records as documents (specifically BSON documents) which are gathered together in collections. A database stores one or more collections of documents.
Databases
In MongoDB, databases hold one or more collections of documents. To select a database to use, in the mongo shell, issue the use <db> statement, as in the following example:
use seleniumdemo
Collections
MongoDB stores documents in collections. Collections are analogous to tables in relational databases.
Documents
A record(row) in MongoDB is a document, which is a data structure composed of field and value pairs. MongoDB documents are similar to JSON objects. The values of fields may include other documents, arrays, and arrays of documents.
MongoDB documents are composed of field-and-value pairs and have the following structure:
{
field1: value1,
field2: value2,
field3: value3,
...
fieldN: valueN
}
What is BSON Type?
BSON is a binary serialization format used to store documents and make remote procedure calls in MongoDB.
The advantages of using documents are:
- Documents (objects) correspond to native data types in many programming languages.
- Embedded documents and arrays reduce the need for expensive joins.
- The dynamic schema supports fluent polymorphism.
Install and Configure MongoDB Community Edition on macOS (for other OS check the references)
MongoDB 4.4 Community Edition supports macOS 10.13 or later. See Supported Platforms for more information.
Install Xcode Command-Line Tools
Homebrew requires the Xcode command-line tools from Apple’s Xcode.
Install the Xcode command-line tools by running the following command in your macOS Terminal:
xcode-select --install
Install Homebrew
macOS does not include the Homebrew brew package by default.
Install brew using the official Homebrew installation instructions.
To install MongoDB, run the following command in your macOS Terminal application:
brew install [email protected]
The installation includes the following binaries:
- The
mongodserver - The
mongosthe sharded cluster query router - The
mongoshell
Install MongoDB Compass
What is MongoDB Compass?
MongoDB Compass is the GUI for MongoDB. Visually explore your data. Run ad hoc queries in seconds. Interact with your data with full CRUD functionality. View and optimize your query performance. Available on Linux, Mac, or Windows. Compass empowers you to make smarter decisions about indexing, document validation, and more.
Install MongoDB Compass
1- Go to this URL and target version and OS then click Download
2- Double click on the .dmg file to install it
3- After you install MongoDB Compass open it from the applications
3- We can add the connection string like the following then click Connect:
mongodb://127.0.0.1:27017/?readPreference=primary&appname=MongoDB%20Compass&ssl=false
This is the default URL of the MongoDB local server
mongodb://127.0.0.1:27017
4- You can create a new Database or edit the existing databases
Part 2: Set up our Project
The Project Prerequisites
- Install Gradle
brew install gradle
- Install IntelliJ IDEA Community Edition
Create a new Gradle project
1- Open IntelliJ IDEA and click Create New Project
2- Select the Gradle project type with Java support
3- Click Next and add the project title and click Finish
4- Wait till the Gradle finish the download and build the project successfully
Install and configure the project prerequisites
5- Open the build.gradle file to add the dependencies (Selenium WebDriver, Webdrivermanager, TestNG, and MongoDB Java Driver)
plugins {
id 'java'
}
group 'org.example'
version '1.0-SNAPSHOT'
repositories {
mavenCentral()
}
dependencies {
testCompile group: 'org.testng', name: 'testng', version: '7.1.0'
compile group: 'org.seleniumhq.selenium', name: 'selenium-java', version: '3.141.59'
testCompile("io.github.bonigarcia:webdrivermanager:3.8.1")
implementation group: 'org.mongodb', name: 'mongo-java-driver', version: '3.12.8'
}
test {
useTestNG() {
}
}
Let’s first use MongoDB Java Driver in our project to try the CRUD operations
The CRUD operations using MongoDB Java Driver
1- Create a Java Class under src/test/java folder like the following image
We will do the following steps:
1- Setup database connection by the following example:
You should create a MongoDB first to be able to connect it through the code
public class MongoDB {
public static void main(String[] args) {
// Connection Setup
MongoClient mongoClient = new MongoClient("localhost", 27017);
MongoDatabase database = mongoClient.getDatabase("demoDB");
MongoCollection<Document> collection = database.getCollection("demoCollection");
}
2- Create a new Document (Record)
The document structure should be
{
username: value1,
password: value2
}
and through the Java Driver will be like the following example:
Document doc = new Document("password", "123456")
.append("username", "moataznabil");
collection.insertOne(doc);
ObjectId id = doc.getObjectId("_id");
System.out.println(id);
We are using collection.insertOne(doc); to insert a new record/document in the Database, then we are retrieving the auto-generated _id from the database to verify that the document inserted correctly.
3- Update existing document:
To update an existing document we need to use collection.updateOne by the ObjectId from the previous insertion operation like the following example:
// Update Operation
collection.updateOne(
eq("_id", new ObjectId(id.toString())),
combine(set("username", "tomsmith")
, set("password", "SuperSecretPassword!"),
currentDate("lastModified")),
new UpdateOptions()
.upsert(true)
.bypassDocumentValidation(true));
doc = collection.find(eq("_id", new ObjectId(id.toString())))
.first();
assert doc != null;
String password = doc.get("password").toString();
String username = doc.get("username").toString();
System.out.println(password);
System.out.println(username);
The we can select it from the database using
doc = collection.find(eq(“_id”, new ObjectId(id.toString()))) .first();
We can also add a new field when updating the document to insert the modification date by using this line currentDate(“lastModified”)),
and verify that the values are updated successfully
4- Retrieve all the documents from the database:
We can use Iterator with find inside the collection to retrieve all the documents and print them in the console to verify the results
// Retrieve All
try (MongoCursor<Document> cursor = collection.find().iterator()) {
while (cursor.hasNext()) {
System.out.println(cursor.next().toJson());
}
}
5- Delete Document
To delete a document we need to use collection.deleteOne and specify the document with the object ID (_Id) like the following example:
// Delete Operation
collection.deleteOne(eq("_id", new ObjectId(id.toString())));
6- Retrieve Documents counts using collection.countDocuments()
// Retrieve Count
System.out.println(collection.countDocuments());
Before running the class, you should comment on the delete function to be able to insert the document into the database
7- Run the class and the final output should be like the following :
604a667eedb7ac0c9a62e56d
SuperSecretPassword!
tomsmith
{"_id": {"$oid": "604a667eedb7ac0c9a62e56d"}, "password": "SuperSecretPassword!", "username": "tomsmith", "lastModified": {"$date": 1615488638088}}
1
8- Run the class again and you will notice that the count will be increased and the returned documents also
604a6787a03cbf56b4bc862f
SuperSecretPassword!
tomsmith
{"_id": {"$oid": "604a6780797bfa69ebffb69a"}, "password": "SuperSecretPassword!", "username": "tomsmith", "lastModified": {"$date": 1615488896816}}
{"_id": {"$oid": "604a6787a03cbf56b4bc862f"}, "password": "SuperSecretPassword!", "username": "tomsmith", "lastModified": {"$date": 1615488903776}}
2
9- Open MongoDB Compass and check the database in a GUI
Part 3: Integrate Selenium WebDriver with MongoDB Database
After we tried the MongoDB Java Driver let’s build our solution with Selenium WebDriver and MongoDB
In our demo, we will use the login page from the Internet website
Create Page Object Class
So first let’s create our page object class
1- Create a new Java class under src/main/java with name LoginPageObject
2- Add the webdriver
WebDriver driver;
3- add a constructor to initialize the driver when creating a new object from the class
public LoginPageObject(WebDriver driver) {
this.driver = driver;
}
4- Add the elements using By keyword like the following code:
By UserName = By.id("username");
By Password = By.id("password");
By LoginBtn = By.className("radius");
5- Implement the function to working with these elements by the following code:
public void typeEmailId(String username) {
driver.findElement(UserName).sendKeys(username);
}
public void typePassword(String password) {
driver.findElement(Password).sendKeys(password);
}
public void clickLoginButton() {
driver.findElement(LoginBtn).click();
}
6- So the final class will be like the following:
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
public class LoginPageObject {
WebDriver driver;
By UserName = By.id("username");
By Password = By.id("password");
By LoginBtn = By.className("radius");
public LoginPageObject(WebDriver driver) {
this.driver = driver;
}
public void typeEmailId(String username) {
driver.findElement(UserName).sendKeys(username);
}
public void typePassword(String password) {
driver.findElement(Password).sendKeys(password);
}
public void clickLoginButton() {
driver.findElement(LoginBtn).click();
}
}
Create BaseTest Class
1- We need first to create a new database and collection for our demo from the MongoDB Compass
2- Create New Java class under src/test/java with the name TestBase
3- At the class, level add the following variables:
public String databaseName = "seleniumdemo" ,
collectionName = "user"
, mongoDBServerURL = "localhost";
public int mongoDBPort = 27017;
public String username = "tomsmith", password = "SuperSecretPassword!";
public String baseURL = "https://the-internet.herokuapp.com/login";
4- Add the WebDriver
protected WebDriver driver;
5- Add a new object from the Login Page Object
LoginPageObject loginPage;
6- Add the following private function to initialize the WebDriver Manager
private void browserSetup(){
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
driver.navigate().to(baseURL);
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(120, TimeUnit.MILLISECONDS);
}
7- Add the following private function to initialize the MongoDB connection and create a new document
private void mongoDBSetup(){
// Setup Mongo DB Connection
MongoClient mongoClient = new MongoClient(mongoDBServerURL, mongoDBPort);
MongoDatabase database = mongoClient.getDatabase(databaseName);
MongoCollection<Document> collection = database.getCollection(collectionName);
assertEquals(databaseName, database.getName());
// Create Document
Document doc = new Document("password", password)
.append("username", username);
collection.insertOne(doc);
ObjectId id = doc.getObjectId("_id");
System.out.println(id);
doc = collection.find(eq("_id", new ObjectId(id.toString())))
.first();
assert doc != null;
password = doc.get("password").toString();
username = doc.get("username").toString();
System.out.println(password);
System.out.println(username);
}
8- Add a void to close the Driver connection with @AfterSuite TestNG annotation
@AfterSuite
public void tearDown() {
if (driver != null) {
driver.quit();
}
}
9- Add a void to initialize all the setup with @BeforeSuite TestNG annotation
@BeforeSuite
public void setUp() {
mongoDBSetup();
browserSetup();
}
10- The final class should be like the following:
import com.mongodb.MongoClient;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import io.github.bonigarcia.wdm.WebDriverManager;
import org.bson.Document;
import org.bson.types.ObjectId;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.chrome.ChromeOptions;
import org.testng.annotations.AfterSuite;
import org.testng.annotations.BeforeSuite;
import java.util.concurrent.TimeUnit;
import static com.mongodb.client.model.Filters.eq;
import static org.testng.Assert.assertEquals;
public class TestBase {
public String databaseName = "seleniumdemo" ,
collectionName = "user"
, mongoDBServerURL = "localhost";
public int mongoDBPort = 27017;
public String username = "tomsmith", password = "SuperSecretPassword!";
public String baseURL = "https://the-internet.herokuapp.com/login";
protected WebDriver driver;
LoginPageObject loginPage;
@BeforeSuite
public void setUp() {
mongoDBSetup();
browserSetup();
}
private void browserSetup(){
WebDriverManager.chromedriver().setup();
driver = new ChromeDriver();
driver.navigate().to(baseURL);
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(120, TimeUnit.MILLISECONDS);
}
private void mongoDBSetup(){
// Setup Mongo DB Connection
MongoClient mongoClient = new MongoClient(mongoDBServerURL, mongoDBPort);
MongoDatabase database = mongoClient.getDatabase(databaseName);
MongoCollection<Document> collection = database.getCollection(collectionName);
assertEquals(databaseName, database.getName());
// Create Document
Document doc = new Document("password", password)
.append("username", username);
collection.insertOne(doc);
ObjectId id = doc.getObjectId("_id");
System.out.println(id);
doc = collection.find(eq("_id", new ObjectId(id.toString())))
.first();
assert doc != null;
password = doc.get("password").toString();
username = doc.get("username").toString();
System.out.println(password);
System.out.println(username);
}
@AfterSuite
public void tearDown() {
if (driver != null) {
driver.quit();
}
}
}
Create our First test with Selenium WebDriver and MongoDB
1- Create a new Java class under src/test/java with the name UserLoginTest and extend the class from the TestBase class
public class UserLoginTest extends TestBase{
}
2- Create a new void with TestNG annotation @Test
@Test
public void userLogin(){
}
3- create a new object from the Login Page Object that added in the TestBase
loginPage= new LoginPageObject(driver);
4- Call our function from the page object class like the following:
loginPage.typeEmailId(username); loginPage.typePassword(password); loginPage.clickLoginButton();
5- Verify that the URL contains the secure keyword which means we already logged in successfully.
System.out.println("Current URL is:" + driver.getCurrentUrl());
Assert.assertTrue(driver.getCurrentUrl().contains("secure"));
6- Run the Test from the green icon beside the Class name or you can use the Gradle command like this example:
./gradlew test --info
7- The output should be the Document ID and the URL printed in the console:
604a72891bd8a701fabe298c SuperSecretPassword! tomsmith Current URL is:https://the-internet.herokuapp.com/secure BUILD SUCCESSFUL in 5s 4 actionable tasks: 1 executed, 3 up-to-date 20:42:05: Tasks execution finished ':cleanTest :test --tests "UserLoginTest"'.
8- You can check the MongoDB database to verify that the document inserted successfully
9- To check the test report you can find the Gradle default report in this path build/reports/tests/test/index.html
10- Open it in your favorite browser and it should be like the following images
Prepare Our Project to Run with CI Server
1- In the TestBase Class we will add a chrome option to run the browser in a headless mode to be like:
private void browserSetup(){
WebDriverManager.chromedriver().setup();
// for CI Purpose
ChromeOptions options = new ChromeOptions();
options.addArguments("--headless");
driver = new ChromeDriver(options);
driver.navigate().to(baseURL);
driver.manage().window().maximize();
driver.manage().timeouts().implicitlyWait(120, TimeUnit.MILLISECONDS);
}
2- Create Script to Install Google Chrome for our CI server:
- In the project create a new folder with name scripts
- Inside this folder create a new shell script file with the name InstallChrome.sh
- Add the following script lines: (we are using a Linux shell script because we will use a Linux machine with GitHub Actions)
#!/bin/bash set -ex wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb sudo apt install ./google-chrome-stable_current_amd64.deb
Part 4: Continuous Integration with GitHub Actions
Introduction to GitHub Actions
GitHub Actions help you automate tasks within your software development life cycle. GitHub Actions are event-driven, meaning that you can run a series of commands after a specified event has occurred. For example, every time someone creates a pull request for a repository, you can automatically run a command that executes a software testing script.
Create GitHub Actions workflow for Our Project
Workflows are custom automated processes that you can set up in your repository to build, test, package, release, or deploy any code project on GitHub. Workflows run in Linux, macOS, Windows, and containers on GitHub-hosted machines You can create a workflow file configured to run on specific events (every pull request, scheduled or manually) There are some limits on GitHub Actions.
GitHub Actions uses YAML syntax to define the events, jobs, and steps. These YAML files are stored in your code repository, in a directory called .github/workflows.
To be able to use GitHub Actions you should do the following steps:
- Create a GitHub Account
- Adding our existing project to GitHub using the command line
- Configure the YAML file for GitHub Action
- Run GitHub Action on every pull request or on schedule
Create a GitHub Account
You can create your free account from this link.
Adding our existing project to GitHub using the command line
Before doing that don’t do the following (Warning from GitHub):
And for more info, you can check also this link.
1- Create a new repository on GitHub. To avoid errors, do not initialize the new repository with README, license, or gitignore files. You can add these files after your project has been pushed to GitHub.
2- Open Terminal.
3- Change the current working directory to your local project.
4- Initialize the local directory as a Git repository.
git init
5- Add the files to your new local repository. This stages them for the first commit.
git add .
6- Commit the files that you’ve staged in your local repository.
git commit -m "add selenium project to GitHub"
7- At the top of your GitHub repository’s Quick Setup page, click to copy the remote repository URL
8- In Terminal, add the URL for the remote repository where your local repository will be pushed.
git remote add origin <REMOTE_URL>
9- Push the changes in your local repository to GitHub.
git push -u origin main
Create GitHub Actions workflow
1- In your repository, create the .github/workflows/ directory to store your workflow files.
2- In the .github/workflows/ directory, create a new file called selenium.yml and add the following code.
name: Selenium Java CI
on: [push]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14.x]
mongodb-version: [4.4]
steps:
- uses: actions/checkout@v2
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
java-version: 1.8
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
- name: Start MongoDB
uses: supercharge/[email protected]
with:
mongodb-version: ${{ matrix.mongodb-version }}
run: |
use seleniumdemo
db.user.insert({username: "moataz", password: "123456"})
- name: Install Google Chrome
run: |
chmod +x ./scripts/InstallChrome.sh
./scripts/InstallChrome.sh
- name: Grant execute permission for gradlew
run: chmod +x gradlew
- name: Build with Gradle
run: ./gradlew test --info
3- Commit these changes and push them to your GitHub repository.
4- Your new GitHub Actions workflow file is now installed in your repository and will run automatically each time someone pushes a change to the repository.
5- Click on the Actions tab to view the history of the builds
Understanding the workflow file
name: Selenium Java CI – Optional the name of the workflow
on: [push] – run automatically each time someone pushes a change to the repository
runs-on: ubuntu-latest – the virutal machine or the hosted machine type
strategy: matrix: can use to run the workflow with a different version in parallel
node-version: [14.x] – The nodejs version
mongodb-version: [4.4] – The MongoDB version
Steps: the workflow steps
– uses: actions/checkout@v2 – checkout the code from the repository-
name: Set up JDK 1.8 – Install and configure Java in the machine
uses: actions/setup-java@v1
with:
java-version: 1.8
– name: Start MongoDB – create Install and setup our MongoDB Database with name seleniumdemo and insert a user into the DB
uses: supercharge/[email protected]
with: mongodb-version: ${{ matrix.mongodb-version }}
run: | use seleniumdemo db.user.insert({username: “moataz”, password: “123456”})
– name: Install Google Chrome
run: | – Install Google Chrome from our script to be able to run our tests
chmod +x ./scripts/InstallChrome.sh
./scripts/InstallChrome.sh
– name: Grant execute permission for gradlew – grant permission to Gradle to be able to run our script from the command line
run: chmod +x gradlew
– name: Build with Gradle
run: ./gradlew test –info
Trigger our tests using GitHub Actions
Our workflow can be triggered automatically each time someone pushes a change to the repository but if we need to run it on schedule we can use the cron syntax for that as the following workflow:
name: Nighlty CI
on:
schedule:
- cron: '* * * * *'
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14.x]
mongodb-version: [4.4]
steps:
- uses: actions/checkout@v2
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
java-version: 1.8
- name: Use Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
- name: Start MongoDB
uses: supercharge/[email protected]
with:
mongodb-version: ${{ matrix.mongodb-version }}
run: |
use seleniumdemo
db.user.insert({username: "moataz", password: "123456"})
- name: Install Google Chrome
run: |
chmod +x ./scripts/InstallChrome.sh
./scripts/InstallChrome.sh
- name: Grant execute permission for gradlew
run: chmod +x gradlew
- name: Build with Gradle
run: ./gradlew test --info
for example in the above workflow, this cron syntax ‘* * * * *’ will run every minute.
And for more info about cron syntax with GitHub Actions check this link.
Thank you for reading and Happy Testing 😉
The GitHub Repository
https://github.com/moatazeldebsy/selenium-github-actions
Article References
- MongoDB
- MongoDB CRUD Operations
- BSON
- Install MongoDB Community Edition on macOS
- Install MongoDB Community Edition on Ubuntu
- Install MongoDB Community Edition on Windows
- MongoDB Java Driver
- Adding an existing project to GitHub using the command line
- GitHub Actions Documentation and here
- Events that trigger GitHub Actions workflows