Timeouts play a vital role in test automation, ensuring test stability and avoiding failures caused by unforeseen delays or unresponsive components. By setting appropriate timeouts, we can control the maximum duration a test can run or wait for a specific condition. This helps maintain test efficiency, detect potential issues promptly, and prevent tests from hanging indefinitely.
When it comes to test automation, pytest is a popular automation testing framework in the Python ecosystem known for its simplicity and extensibility. In the context of pytest, it is crucial to handle timeouts effectively to ensure smooth test execution and reliable results.
In this Python tutorial on pytest timeouts, we will dive deeper into understanding pytest timeouts, including their configurations, handling exceptions, strategies, and best practices.
So, let’s get started.
Timeout is a crucial concept that revolves around the maximum allowable time for a specific operation to complete successfully. It safeguards against potential bottlenecks and performance issues that can disrupt the smooth execution of test cases.
They are especially important in scenarios where test scripts interact with web elements, make network requests, or wait for specific conditions to be met. They ensure that the automated tests do not hang indefinitely, preventing a single misbehaving test from causing a domino effect on the entire test suite.
A timeout is generally set in seconds, indicating the maximum duration a test step can wait for a certain condition or action.
For instance, when waiting for a web page to load, a timeout ensures that the script doesn’t wait indefinitely if the page takes longer to load than expected. Suppose the specified action isn’t complete within the defined timeout period. In that case, the automation script will either proceed with an alternative action or mark the test case as failed, depending on the implementation. This helps maintain test efficiency, detect potential issues promptly, and prevent tests from hanging indefinitely.
Explore Jest tutorial, your go-to guide for mastering Jest testing. Run Jest automation tests in parallel across multiple browser and OS combinations.
If an alternative action is not set, a timeout error is thrown:
Timeouts are especially critical when dealing with interactions with external resources such as web services, APIs, databases, or UI elements. A single slow or unresponsive component can significantly impact the overall test execution time and result in flaky test outcomes.
Page load delays stem from factors like server load and network constraints, potentially resulting in timeouts when fixed wait times clash with extended load times.
Element identification via locators like CSS Selectors and XPath can lead to timeouts if elements are missing or rendering is sluggish.
Asynchronous operations such as AJAX requests can introduce delays in content updates, risking timeouts if not effectively managed.
Network latency, caused by slow connectivity, can result in timeouts while waiting for server responses.
Fluctuating UI load times across varying components may introduce timing inconsistencies, leading to prolonged script waits or early timeouts.
Dynamic content generation, common in web pages, can trigger timeouts if elements aren’t rendered promptly post-initial load.
Synchronization hitches between test script execution and webpage rendering can be a timeout culprit, especially in the absence of well-implemented explicit waits for asynchronous operations.
By incorporating timeout handling into our test automation strategies, we can mitigate these challenges and enhance the reliability and efficiency of our test suites.
Understanding pytest Timeouts
Pytest provides a built-in mechanism for handling timeouts during test execution. In pytest, a timeout refers to the maximum duration a specific test or test step is expected to complete. If the execution time exceeds this threshold, pytest considers it a timeout failure and terminates the test with an appropriate exception. This helps prevent tests from running indefinitely and ensures timely feedback on test failures caused by long-running operations.
It’s important to note that the timeout value specified in pytest represents an upper limit and does not guarantee the exact execution time. Various factors, such as system load, network latency, or the performance of external dependencies, can affect the actual test execution time. Therefore, timeouts should be set strategically, allowing flexibility while preventing excessive delays.
Consider the following sample code that results in a timeout exception:
import requests
def test_api_response():
# Set a very small timeout to provoke the Timeout exception
response = requests.get('https://reqres.in/api/users', timeout=0.0001)
# Call the function to execute it and provoke the exception
test_api_response()
The above code imports the requests module, which is commonly used in Python for making HTTP requests. Within the code, there is a function named test_api_response(). This function is designed to send a GET request to the URL ‘reqres.in/api/users’. The timeout parameter for this request is set to a minimal value (0.0001 seconds) to provoke a timeout exception intentionally.
The function will raise an exception if the server doesn’t respond within that minuscule duration. After defining the function, it is immediately called, executing the request and likely resulting in the exception, as mentioned earlier.
In this XCUITest tutorial, learn about XCUITest framework and its benefits for mobile automation testing. Take a look at how XCUITest works and see how to use it to test your mobile applications.
Differentiating Between Test Execution Time and Timeout
While the test execution time represents the actual duration a test takes to run, the timeout value is a safeguard to prevent tests from running indefinitely or waiting too long for a specific condition. Understanding this distinction is crucial to handle timeouts in your test automation workflow effectively.
Consider a scenario where you have a test that interacts with a web page and waits for a specific element to appear. The test execution time includes navigating the page, interacting with elements, and asserting the expected conditions. On the other hand, the timeout value defines the maximum time the test is allowed to wait for the element to appear.
By separating the concept of execution time and timeout, you can distinguish between tests that legitimately take longer to execute and tests that may indicate potential issues due to excessive wait times. This differentiation helps in identifying and addressing bottlenecks or flaky tests more efficiently.
Common Scenarios for Handling Timeout Error
Timeouts are particularly useful in various scenarios encountered during test automation. Let’s explore a few common situations where timeouts can significantly improve the reliability and stability of your test suite:
- Waiting for element visibility
When interacting with web elements or performing automated UI testing, timeouts are essential to avoid waiting indefinitely for elements to become visible or interactable. Setting appropriate timeouts ensures that tests proceed promptly even if expected elements do not appear within a reasonable timeframe.
The provided code demonstrates a test case for successful login functionality using Selenium with Python.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
def test_login_success():
# Wait for the login button to be visible within 10 seconds
login_button = WebDriverWait(driver, 10).until(
EC.visibility_of_element_located((By.ID, 'login-button'))
)
# Perform login actions
# ...
The test begins by utilizing the WebDriverWait class from Selenium’s support.ui module to wait for the login button to become visible on the webpage. The WebDriverWait class allows us to set a maximum timeout of 10 seconds, ensuring that the test doesn’t proceed until the login button appears.
The until() method is then used with EC.visibility_of_element_located() to specify the expected condition for the login button’s visibility, identified by its ID attribute, ‘login-button’.
Upon successful visibility of the login button, the test proceeds to the next steps, which involve performing login actions specific to the application being tested.
This approach of using explicit waits with WebDriverWait and expected_conditions ensures that the test case executes reliably and stably, allowing ample time for the login button to appear before proceeding with the subsequent actions. If the condition is not satisfied and a timeout occurs, a TimeoutError is thrown, as shown in the previous section.
This is a step-by-step solution to address this challenge/UI-related timeouts:
Leverage explicit waits provided by Selenium WebDriver to wait for elements with specified timeout values. Use expected conditions like element visibility, element presence, or custom conditions to handle timeouts.
Implement dynamic waits using WebDriverWait and ExpectedConditions to wait until a specific condition is met or a timeout occurs, whichever happens first.
Adjust the polling interval of explicit waits to balance responsiveness and resource consumption.
- Network requests
When performing API testing or interacting with web services, timeouts prevent excessive delays caused by slow network responses. By setting appropriate timeouts, you can ensure that tests do not hang indefinitely due to unresponsive endpoints or network issues.
import requests
def test_api_response():
# Set a timeout of 5 seconds for the API request
response = requests.get('https://api.example.com', timeout=5)
# Perform assertions on the response
# ...
The given code exemplifies a test case that validates the API response using Python’s requests library. The test starts by importing the requests module, commonly used for making HTTP requests in Python. Within the test, a GET request is sent to the URL ‘https://api.example.com‘, with a specified timeout of 5 seconds.
The timeout parameter is utilized to set a maximum time for the API request to receive a response. Suppose the response is not received within the specified timeout duration. A TimeoutError exception is raised, enabling the test to handle scenarios where the API might take longer than expected to respond.
Following the API request, the test performs assertions on the response to validate its content, status, or any other relevant information.
This is a step-by-step solution to address this challenge/network-related timeouts:
Set appropriate timeouts for network requests using libraries like requests or Selenium WebDriver’s implicit and explicit waits.
Implement retry mechanisms using libraries like retrying or custom retry logic to handle transient network failures.
Incorporate logging and reporting mechanisms to capture and analyze network-related timeout failures for further investigation.
Selenium is an open-source framework designed for automating web browsers for testing web applications. Delve into its architecture, benefits, and more through our detailed Selenium tutorial for enhanced cross-browser testing.
- Database operations
When executing tests involving database interactions, timeouts are useful to prevent tests from waiting indefinitely for long-running queries or transactions. Although many databases have built-in mechanisms to handle timeouts, these built-in timeouts might not always align with the desired timing for a test.
Implementing your timeout in a test framework ensures that tests don’t exceed the allocated time, regardless of the database’s settings. Setting appropriate timeouts allows you to handle scenarios where the database connection becomes unresponsive, or queries take longer than expected.
In the below code snippet, you need first to install the psycopg2 library using the pip command:
pip install psycopg2
import psycopg2
from psycopg2 import sql
def test_database_query():
# Connect to the database
conn = psycopg2.connect(database="mydb", user="myuser", password="mypassword", host="localhost", port="5432")
# Set the timeout for the database query to 10 seconds
conn.set_session(sql.SQL("SET statement_timeout TO '10s'"))
# Execute the query
# ...
The provided code demonstrates a test scenario involving a database query using the psycopg2 library in Python. First, the psycopg2 module is imported, which is widely used for connecting to and working with PostgreSQL databases. Additionally, the sql module is imported to help build SQL queries.
In the test, the psycopg2.connect() function establishes a connection to the PostgreSQL database named “mydb”. The connection parameters include the database name, username — “myuser“, password “mypassword“, host “localhost“, and port “5432.”
Following the database connection, the test sets a specific timeout for the database query by executing the SQL command “SET statement_timeout TO ’10s‘”. This statement configures the database session to have a timeout of 10 seconds, ensuring that any query executed on this session will be canceled if it takes longer than the specified timeout.
This is a step-by-step solution to address this challenge/long-running database operations:
Optimize your database queries by ensuring proper indexing, minimizing unnecessary operations, and employing query optimization techniques.
Analyze the database schema and design to ensure efficient data retrieval and updates.
Set appropriate database query timeouts to prevent tests from waiting indefinitely.
Consider distributing tests across multiple processes or threads to parallelize database operations and reduce execution time.
Selenium is an open-source framework designed for automating web browsers for testing web applications. Delve into its architecture, benefits, and more through our detailed Selenium guide for enhanced cross-browser testing.
What is pytest-timeout?
pytest-timeout enables you to control the duration of your tests, allowing you to address instances where a test’s prolonged execution may indicate underlying issues within your system or the test script itself. With pytest-timeout, you can set how long a test is allowed to run. If it goes beyond that, you get an alert.
How to install pytest-timeout?
Installing pytest-timeout is too simple. Just ensure you have pytest installed first, as pytest-timeout works with it. Open your terminal and type this command:
pip install pytest-timeout
Once installed, you can start using it for better test control.
Configuring Timeout in pytest
Pytest provides two primary ways to configure timeouts:
Global Configuration
Per-test Configuration
Global Configuration
With Global Configuration, you can set a default timeout value that applies to many test cases within your test suite, which is a collection of related test cases designed to validate a specific set of functionalities within a software application. This global configuration provides a consistent timeout threshold across the test suites and simplifies the process of specifying timeouts for individual tests.
If you want to set a timeout value that should be used by all the test cases in all the test suites, you should use the Global Configuration.
The big benefit is a simple way to set a timeout that applies to all test cases.
Setting Timeout at Global-level in pytest
Once pytest-timeout is installed, you can set the global timeout value by creating a pytest.ini file at the root of your project and adding the following configuration:
[pytest]
timeout = 10
In this example, we set the global timeout value to 10 seconds. This means that if a test exceeds 10 seconds of execution time, it will be considered a timeout failure.
WebDriver is a remote programming interface that can be used to control, or drive, a browser either locally or on a remote machine. Learn more in this complete Selenium WebDriver Tutorial.
Per-test Configuration
Per-test Configuration allows you to override the global timeout value for specific tests. This gives you granular control over individual test timeouts, allowing you to tailor the timeout based on the specific requirements of each test case.
If you want to set a specific timeout value that should be used in a specific test case, for example, when you know it costs more time to run, you should use the Per-test Configuration.
The main benefit is that you can keep a small timeout set in the Global Configuration for all test cases, and for specific tests, you should set bigger values, keeping your test suite quicker to run.
Setting Timeout at Per-test-level in pytest
While the global timeout provides a convenient way to set a default value for all tests, there are cases where certain tests require different timeout thresholds. Pytest allows you to override the global timeout on a per-test basis.
To override the global timeout, you can utilize the timeout marker @pytest.mark.timeout provided by the pytest-timeout plugin. Let’s consider an example where we have a test that requires a longer timeout:
import pytest
import time
@pytest.mark.timeout(20)
def test_long_running_operation():
# Perform a long-running operation
time.sleep(2) # Simulate a time-consuming operation
# ...
In this example, we use the @pytest.mark.timeout(20) decorator to specify a timeout of 20 seconds for the test_long_running_operation() test. This overrides the global timeout value defined in pytest.ini (as shown in the previous sample) for this test case. Inside the test function, a simple time.sleep(2) operation is used to simulate a time-consuming action.
Exploring timeout units (seconds, milliseconds, etc.)
Pytest allows you to specify timeouts using different time units, including seconds, milliseconds, minutes, or even human-readable formats. This flexibility lets you express timeouts most conveniently for your test automation needs.
By default, pytest interprets timeout values as seconds. However, you can use specific suffixes to specify timeouts in different units (e.g., 1s, 100ms, 2m, etc.).
import time
import pytest
# Test function with different timeouts specified in various time units
@pytest.mark.timeout(1) # 1 second timeout (default unit: seconds)
def test_one_second_timeout():
time.sleep(2) # Simulate a time-consuming operation
@pytest.mark.timeout(100, 'ms') # 100 milliseconds timeout
def test_100_milliseconds_timeout():
time.sleep(0.2) # Simulate a time-consuming operation
@pytest.mark.timeout(2, 'm') # 2 minutes timeout
def test_two_minutes_timeout():
time.sleep(120) # Simulate a time-consuming operation
In this code, we used the @pytest.mark.timeout decorator to specify different timeouts for each test function. The default unit for timeout is seconds, so the test_one_second_timeout has a timeout of 1 second. We also explicitly specify timeouts in milliseconds and minutes for test_100_milliseconds_timeout and test_two_minutes_timeout, respectively.
When running these tests using pytest, the specified timeouts will be applied. If a test exceeds its specified timeout duration, pytest will raise a TimeoutError exception, indicating that the test has timed out.
Selenium WebdriverIO is Javascript based test automation framework built over nodeJs. Learn with this guide how to use webdriverIO to perform web automation testing.
Using timeout methods through the command line
Another very interesting way to use timeout is through the command line. pytest-timeout allows you to send the desired timeout and the test execution command. Let’s learn how to use the command line to set timeouts and the methods that can be used for handling timeouts.
Signal
The signal method uses system signals to handle timeouts. It’s like sending a quick message to a test that’s taking too long. But this way doesn’t work on all computers and has some limits.
To use the signal method with pytest-timeout, just add this when you run your tests:
pytest — timeout=<value> — timeout-method=signal
Just replace < value > with the timeout value in seconds.
Thread
The thread method uses threads in Python. It’s like starting a helper that watches your test. If the test goes on too long, the helper stops the execution. This prevents tests from running forever.
To use the thread method, use the below command:
pytest — timeout=<value> — timeout-method=thread
Again, replace < value > with the time you want in seconds.
Handling pytest Timeout Exceptions
When a timeout occurs during test execution in pytest, it raises a TimeoutError exception. This exception essentially serves as an alarm, indicating that a particular test has exceeded the predefined time limit for completion. Timeouts are valuable in testing because they help prevent tests from running indefinitely or hanging indefinitely due to unexpected issues.
When a TimeoutError is raised, it stops the test and provides valuable feedback to testers and developers, pointing out that something might be amiss, whether it’s a performance problem in the application being tested or an issue with the test case itself. This exception acts as a safety mechanism, ensuring that tests don’t run indefinitely and helping maintain the efficiency and reliability of the testing process.
In this section of the blog on pytest timeouts, we will look into different ways to handle pytest timeout exceptions.
Recognizing timeout-related exceptions in pytest
When a test exceeds its specified timeout threshold, pytest raises a TimeoutError exception, as shown in the previous sections. This exception indicates that the test execution time has surpassed the allowed duration, resulting in a pytest timeout failure.
To handle timeout-related exceptions effectively, it’s essential to catch and respond to the TimeoutError appropriately. By doing so, you can customize the behavior, perform cleanup actions, or take specific steps based on the nature of the pytest timeout failure.
Perform native app testing on real mobile test devices online. Test on Android and iOS devices with LambdaTest. Run automated or manual native app tests. Start testing for free today!
Handling timeouts gracefully using pytest features
Pytest offers several features that can help you handle timeouts gracefully and improve the stability of your test suite. Let’s explore some of these features:
- Test-specific timeout handling
Pytest allows you to specify different timeout values for individual tests, enabling you to fine-tune the pytest timeout thresholds based on the nature of each test case. By setting appropriate timeouts, you can ensure that tests are completed within reasonable durations and avoid false-positive or false-negative results.
- Retry mechanisms for flaky tests
In some cases, a test may occasionally fail due to intermittent issues, such as network glitches or temporary unavailability of external resources. Pytest provides plugins like pytest-rerunfailures or pytest-flaky that automatically allow you to retry flaky tests. By retrying tests that fail due to timeouts, you can increase the chances of obtaining reliable test results.
- Logging and reporting timeout failures
It’s crucial to log and report them adequately to facilitate debugging and analysis. Pytest provides built-in mechanisms for capturing test execution details, including timeouts. By leveraging reporting plugins like pytest-html, you can generate detailed HTML reports that include timeout information, enabling easier identification and analysis of timeout-related issues.
Strategies for Mitigating Timeout Issues
Timeouts can be a common challenge in test automation, but there are strategies you can employ to mitigate timeout issues and improve the stability of your tests.
Optimizing test execution time
One of the primary ways to mitigate timeout issues is to optimize the execution time of your tests. By reducing unnecessary test steps and streamlining your test automation process, you can minimize the likelihood of timeouts occurring.
Consider the following strategies:
- Reducing unnecessary test steps
Analyze your test cases and eliminate any redundant or unnecessary steps contributing to longer execution times. Focus on essential actions that directly validate the functionality being tested.
- Leveraging test fixtures and test data management
Utilize test fixtures to set up and tear down common test data or test environment configurations like timeout.
Test fixtures refer to the predefined set of conditions and data necessary to create a controlled testing environment. By using test fixtures, you can establish a consistent starting point for your tests. This includes setting up common test data or configuring the testing environment to replicate real-world scenarios.
Reusing fixtures across multiple tests offers several benefits, such as time efficiency, consistency, and maintenance.
Dealing with slow external dependencies
External dependencies, such as web services or databases, can significantly impact the overall test execution time and lead to timeouts.
Here are some strategies to handle slow external dependencies effectively:
- Mocking or stubbing slow components
Instead of relying on real external dependencies during every test, use mocking or stubbing techniques to simulate their behavior. This allows you to bypass the potential slowdown caused by these dependencies and enables faster and more reliable tests.
- Asynchronous testing techniques
If you’re dealing with asynchronous operations, consider using asynchronous testing techniques such as asyncio or async/await syntax.
These techniques allow you to manage long-running operations without blocking the execution flow, preventing potential timeouts.
- Use a Page Load Strategy in Selenium
In Selenium, various page loading strategies determine how the WebDriver handles resource loading and availability while automating web interactions. These strategies — Normal, Eager, and None — influence the WebDriver’s behavior during different stages of page loading.
Normal Strategy: This is the default strategy. It instructs the WebDriver to wait until all resources on a web page, including images, stylesheets, scripts, and more, have finished downloading before proceeding with interactions.
Eager Strategy: In this strategy, the WebDriver waits for the Document Object Model (DOM) to be accessible. This means that even though the core structure of the page is ready, other resources, such as images, may still be loading in the background.
None Strategy: Opting for the None strategy means the WebDriver doesn’t wait for any resources to load before moving forward with interactions.
Upload your app on LambdaTest for android device testing online. Make the most of android application testing with official android emulators or real android devices on LambdaTest.
Identifying and handling long-running tests
Long-running tests can be a significant source of timeout issues. It’s essential to identify and handle them appropriately to maintain efficient and reliable test execution.
Consider the following strategies:
- Test profiling and performance analysis
Analyze your test suite to identify tests that consistently take longer to execute. Profile these tests to identify potential bottlenecks and optimize their execution time. This may involve optimizing database queries, improving algorithmic efficiency, or employing parallelization techniques.
- Distributing tests across multiple processes/threads
If your test framework supports parallel test execution, consider distributing tests across multiple processes or threads. Parallelization can help reduce the overall test suite execution time and alleviate potential timeout issues caused by long-running tests.
Handling Timeouts in Selenium Automation Testing
In this section, we will apply some of the knowledge of this blog about handling pytest timeouts in some Selenium automation tests for both Global and Per-test levels.
Venturing into the domain of parallel Selenium automation testing brings forth the opportunity to expedite testing efforts by effectively utilizing concurrency. By simultaneously executing multiple test cases, we can significantly reduce overall testing time while maximizing the efficiency of our test suite.
You can discover how to conduct Python automation testing for your websites or web applications using a cloud-based grid like LambdaTest. LambdaTest is an AI-powered test orchestration and execution platform that lets you run manual and automated tests on over 3000+ real browsers, devices, and OS combinations at scale.
Don’t miss out on thе latеst tutorials on automation tеsting! Subscribe to thе LambdaTеst YouTubе Channеl for tutorials on Sеlеnium tеsting, Playwright testing, and more.
Visit our documentation to get started with Selenium Python testing.
To perform Selenium Python testing on the LambdaTest cloud grid, you should use the capabilities to configure the environment where the test will run. In this blog on pytest timeout, we will run the tests in the following environments:
Environment 1: Browser: Chrome / OS: Windows 11
Environment 2: Browser: Firefox / OS: macOS Ventura
Test your native app and website on real iOS and Android devices hosted on the cloud. LambdaTest is a convenient, cost-effective and centralised solution for running realtime and Automated test on cloud device testing.
We are going to execute the below test scenarios:
Test Scenario 1
Precondition:
- Global timeout in pytest.ini is set to 30 seconds.
Steps:
1. Open the Simple Form Demo page in the LambdaTest Playground.
2. Fill in the first field with a message.
3. Click the Get Checked Value button.
4. Check that the typed message is shown to the user on the right side.
Test Scenario 2
Precondition:
- Global timeout in pytest.ini is set to 300 seconds.
Steps:
1. Open the Simple Form Demo page in the LambdaTest Playground.
2. Fill in the first field with a message.
3. Click the Get Checked Value button.
4. Check that the typed message is shown to the user on the right side.
Test Scenario 3
Precondition:
Global timeout in pytest.ini is set to 30 seconds.
Test case timeout using pytest-timeout decorator set to 300 seconds.
Steps:
1. Open the Simple Form Demo page in the LambdaTest Playground.
2. Fill in the first field with a message.
3. Click the Get Checked Value button.
4. Check that the typed message is shown to the user on the right side.
Setting up the Environment
Before embarking on the journey of parallel testing, laying a solid foundation for the testing environment is paramount. Correctly setting up the testing environment is a pivotal step that guarantees a smooth coding experience and maximizes testing capabilities.
To optimize testing efforts and enhance productivity, let’s explore the essential steps that should be taken before diving into parallel testing. Following these crucial actions can pave the way for a seamless and efficient testing process, ensuring successful outcomes in our endeavors.
Step 1: Download and Install Python
Begin by installing Python, if not already installed, from the official Python website.
Step 2: Install Selenium and pytest libraries
Once Python is installed, use the Python package manager, pip, to install Selenium and pytest just by running the following command:
pip install -r requirements.txt
Requirements.txt contains the dependencies that we want to install.
After running, you can see the below output:
Perform native app testing on real mobile devices testing online free . Test on Android and iOS devices with LambdaTest. Run automated or manual native app tests. Start testing for free today!
Step 3: Download and Install Visual Studio Code.
Step 4: Configure pytest in Visual Studio Code
To finish the configuration, we need to say to Visual Studio Code that pytest will be our Test Runner, so you can do this following the below instructions:
1. Create a folder for your project (in our example python-pytest-timeout).
2. Open the project folder in Visual Studio Code.
3. Open the command palette (menu View > Command Palette), and type “Configure Tests”.
4. Select pytest as the test framework.
5. Select the root directory option.
You must also prepare the LambdaTest capabilities code to be inserted in our test script.
You can generate the capabilities code from the LambdaTest Capabilities Generator.
Then, you should get the “Username” and “Access Token” from your account under Password & Security and set it as an environment variable.
With everything carefully organized and all the requirements met, we can confidently and eagerly begin. The time has come to harness the full potential of our knowledge and experience as we delve into the fascinating testing world. Without any delay, let’s dive into the depths of parallel testing and embark on this thrilling adventure together.
Implementation of Parallel Testing on LambdaTest
First of all, let’s create the project structure:
Implementation
[WEBSITE]
url = https://www.lambdatest.com/selenium-playground/simple-form-demo
[LOGIN]
username = LT_USERNAME
access_key = LT_ACCESS_KEy
[CLOUDGRID]
grid_url = hub.lambdatest.com/wd/hub
build_name = Python Timeout Build
test_name = Test Case X
w3c = True
browser_version = latest
selenium_version = 4.8.0
[ENVWIN]
platform = Windows 11
browser_name = Chrome
[ENVMAC]
platform = MacOS Ventura
browser_name = Firefox[WEBSITE]
url = https://www.lambdatest.com/selenium-playground/simple-form-demo
[LOGIN]
username = LT_USERNAME
access_key = LT_ACCESS_KEy
[CLOUDGRID]
grid_url = hub.lambdatest.com/wd/hub
build_name = Python Timeout Build
test_name = Test Case X
w3c = True
browser_version = latest
selenium_version = 4.8.0
[ENVWIN]
platform = Windows 11
browser_name = Chrome
[ENVMAC]
platform = MacOS Ventura
browser_name = Firefox
[pytest]
timeout = 30
from selenium import webdriver
from selenium.webdriver.common.by import By
import pytest
import os
import configparser
import time
# Load the configuration file
config = configparser.ConfigParser()
config.read('config/config.ini')
@pytest.fixture(params=["test-for-win", "test-for-mac"],scope="class")
def driver(request):
username = os.getenv("LT_USERNAME")
config.set('LOGIN', 'username', username)
accessKey = os.getenv("LT_ACCESS_KEY")
config.set('LOGIN', 'access_key', accessKey)
username = config.get('LOGIN', 'username')
accessKey = config.get('LOGIN', 'access_key')
gridUrl = config.get('CLOUDGRID', 'grid_url')
if request.param == "test-for-win":
web_driver = webdriver.ChromeOptions()
platform = config.get('ENVWIN', 'platform')
browser_name = config.get('ENVWIN', 'browser_name')
if request.param == "test-for-mac":
web_driver = webdriver.FirefoxOptions()
platform = config.get('ENVMAC', 'platform')
browser_name = config.get('ENVMAC', 'browser_name')
lt_options = {
"user": config.get('LOGIN', 'username'),
"accessKey": config.get('LOGIN', 'access_key'),
"build": config.get('CLOUDGRID', 'build_name'),
"name": config.get('CLOUDGRID', 'test_name'),
"platformName": platform,
"w3c": config.get('CLOUDGRID', 'w3c'),
"browserName": browser_name,
"browserVersion": config.get('CLOUDGRID', 'browser_version'),
"selenium_version": config.get('CLOUDGRID', 'selenium_version'),
"networkThrottling": "Regular 3G"
}
options = web_driver
options.set_capability('LT:Options', lt_options)
url = f"https://{username}:{accessKey}@{gridUrl}"
driver = webdriver.Remote(
command_executor=url,
options=options
)
yield driver
driver.quit
def test_demo_form_using_global_timeout(driver):
driver.get(config.get('WEBSITE', 'url'))
# Find an input element by its ID and enter text
input_element = driver.find_element(By.ID, "user-message")
input_element.send_keys("This is a timeout text!")
# Find an element by its ID and click on it
element = driver.find_element(By.ID, "showInput")
element.click()
# Find an element by its ID and extract its text
element = driver.find_element(By.ID, "message")
assert element.text == "This is a timeout text!"
Same as test_cloud_grid_1.py, just with one test function shown below.
def test_demo_form_using_global_timeout_exceeding(driver):
driver.get(config.get('WEBSITE', 'url'))
# Find an input element by its ID and enter text
input_element = driver.find_element(By.ID, "user-message")
input_element.send_keys("This is a timeout text!")
# Simulate a time-consuming operation
# Global timeout is set to 30 seconds
time.sleep(31)
# Find an element by its ID and click on it
element = driver.find_element(By.ID, "showInput")
element.click()
# Find an element by its ID and extract its text
element = driver.find_element(By.ID, "message")
assert element.text == "This is a timeout text!"
Perform native app testing on real mobile online device testing. Test on Android and iOS devices with LambdaTest. Run automated or manual native app tests. Start testing for free today.
Code Walkthrough
config.ini
This file contains configuration settings in the INI file format. It includes sections for different settings, such as website URL, login credentials, cloud grid options, and test environment details for Windows and macOS.
The [WEBSITE] section contains the website URL to be tested.
The [LOGIN] section includes the username and access key placeholders for authentication. These values will be read from environment variables and set in the driver fixture later.
The [CLOUDGRID] section holds various settings related to the cloud grid, such as the URL, build name, test name, and other options.
The [ENVWIN] and [ENVMAC] sections store environment-specific details like the platform (Windows or macOS) and browser name (Chrome or Firefox) to be used for testing.
pytest.ini
This file configures pytest options for test execution. In this case, it sets a global test timeout of 30 seconds for all test functions.
In this case, the timeout option is set to 30 seconds. This means that all test functions will have a global timeout of 30 seconds. Any test that takes longer than 30 seconds will be terminated and marked as a failure.
This Python test script defines two test functions that use the pytest fixture named driver to perform web testing using Selenium WebDriver with different configurations.
The test_cloud_grid_1.py script begins with import statements for necessary modules, including selenium.webdriver for web testing, pytest for testing features, os for environment variables, configparser for reading the configuration file, and time for time-related operations.
The fixture sets up a Selenium WebDriver based on the test environment. It reads configuration settings from config.ini to get username, access key, grid URL, platform, and browser name. It also sets specific capabilities required for remote WebDriver execution. The fixture is scoped at the class level, which will be executed once for each test class.
The driver fixture is parametrized with two values: “test-for-win” and “test-for-mac”. Any test function using this fixture will be executed twice, once for each parameter value. In this case, the fixture will create a WebDriver instance for both Windows and macOS platforms. The fixture sets the desired capabilities for the WebDriver instance using options.set_capability() to include the LambdaTest-specific options stored in the lt_options dictionary.
One important feature is the networkThrottling property set to “Regular 3G” to make the connection slower and simulate an environment with a bad network connection.
It then constructs the URL to connect to the LambdaTest hub with the appropriate username, access key, and grid URL. The WebDriver instance is created using webdriver.Remote() by passing the command_executor (the LambdaTest hub URL) and the options with desired capabilities.
The yield driver statement allows the test functions using this fixture to access the WebDriver instance and use it for testing. After completing the test function(s), the fixture will execute the driver.quit statement to close the WebDriver and release any associated resources.
test_demo_form_using_global_timeout_not_exceeding.py
This test navigates to the URL specified in the configuration file using driver.get().
Moving forward, we start locating the elements based on the web page’s structure.
It finds an input element by its ID (“user-message”) and enters the text “This is a timeout text!” into it.
After that, it finds another element by its ID (“showInput”) and clicks on it.
Finally, it finds an element by its ID (“message”) and asserts that its text is equal to “This is a timeout text!”.
Let’s run this code. Use the below command:
pytest test_global_timeout.py
As a result, this test function will raise a pytest.TimeoutError because it takes longer than the allowed global timeout to complete, that is 30 seconds.
Let’s now change the content of pytest.ini to set the timeout to 300 seconds.
[pytest]
timeout = 300
Let’s rerun the same test using the pytest command:
pytest test_global_timeout.py
Now, the time to run the test does not achieve the timeout limit, the tests are correctly executed, and are now passing.
This Python test script is almost identical to test_global_timeout.py.
First, you should set the global timeout back to 30 seconds.
[pytest]
timeout = 30
Consider that the test_demo_form_using_per_test_timeout content is the same as the test_demo_form_using_global_timeout.
If we run the test now, the same pytest.TimeoutError will happen.
However, we can now mark this test function with @pytest.mark.timeout(300), which only sets a specific timeout of 300 seconds for this test function.
When running the below command
pytest test_per_test_timeout.py
We have the following results:
As a result, this test function will not raise a pytest.TimeoutError anymore because it takes less than the allowed per-test timeout to complete.
The two tests inside test_per_test_timeout.py passed in the four executions, one for Windows and one for Mac.
We can also see this in the LambdaTest Dashboard.
Overall, this code demonstrates how to handle Python pytest timeout in test cases where the global timeout and the test timeout were exceeded and were not, using a Python and Selenium test automation scenario executed in parallel in multiple environments using the LambdaTest platform.
Test your native app and website on real iOS and Android devices hosted on the cloud. LambdaTest is a convenient, cost-effective and centralised solution for running realtime and Automated test on real device cloud.
Best Practices for Timeout Management
To ensure effective timeout management in your test automation workflow, following best practices and establishing a systematic approach is essential.
- Applying timeouts judiciously
It’s crucial to apply timeouts judiciously, considering the specific requirements and characteristics of each test case.
Avoid setting overly aggressive timeouts that may lead to false-positive failures or overly permissive timeouts that mask real issues. Aim for a balance that allows sufficient time for the expected behavior while still detecting potential problems within a reasonable timeframe.
- Setting realistic timeout thresholds
When setting timeout thresholds, strive for realism based on factors such as network conditions, system performance, and the nature of the application under test.
Collaborate with developers and stakeholders to establish acceptable timeout ranges that align with user expectations and performance goals.
- Regularly reviewing and adjusting timeouts
Timeouts should not be set in stone. Regularly review and reassess the effectiveness of your timeout values based on test results, feedback, and changes in the system under test.
Adapt the timeouts to ensure they remain relevant and aligned with evolving requirements.
- Collaborating with developers to address performance issues
Timely collaboration with developers is essential to address underlying performance issues that may lead to timeouts. When encountering persistent timeout failures, work with the development team to identify and resolve performance bottlenecks, optimize queries, or improve system responsiveness.
This collaborative effort will contribute to a more stable and reliable test automation ecosystem.
Are you a developer or tester aspiring to become a skilled Python automation tester? Don’t worry! You can achieve this goal by attempting the Selenium Python 101 certification program. This certification will elevate your Python proficiency and provide a strong basis for effectively applying Selenium Python in your testing endeavors, enabling you to thrive in your automation testing profession.
Want to ensure your website is optimized for mobile traffic? Our comprehensive guide to test website on real mobile devices covers everything you need to know.
Conclusion
In this Selenium pytest tutorial, we explored different ways of handling timeouts in pytest, understanding the concept of timeouts and their significance in test automation. We learned how to configure pytest timeouts at global and per-test levels, allowing us to set timeout thresholds that suit our specific needs. We discussed strategies for gracefully handling pytest timeout exceptions and explored best practices for timeout management.
We explored real-world examples and use cases and implemented a Python with Selenium test automation scenario executed in parallel in multiple environments using the LambdaTest platform.
Timeout management is an ongoing process that requires continuous evaluation, collaboration inside the team, and adaptation to changing requirements. As you gain experience handling timeouts, you will develop strategies and techniques to overcome these challenges effectively.
Frequently Asked Questions (FAQs)
Does Pytest have a timeout?
Yes, pytest provides a built-in mechanism for setting timeouts on test functions using the @pytest.mark.timeout decorator. This feature lets you specify a maximum execution time for individual test cases. If a test case exceeds the specified timeout, pytest will automatically mark it as failed. This is useful for preventing tests from running indefinitely and for handling scenarios where a test might hang or take too long to complete.
How do you give timeout in Python?
In Python, you can implement timeouts for various operations, such as function calls or blocking I/O, using several approaches. Here are some common methods:
Using the timeout-decorator Library:
Using the threading Module.
Using the multiprocessing Module.
Using Third-Party Libraries.
How do you rerun failed test cases in Pytest?
In pytest, you can rerun failed test cases using the –reruns and –reruns-delay command-line options. These options allow you to specify the number of times you want to rerun failed tests and the delay between each rerun.
Want to ensure your website is optimized for mobile traffic? Our comprehensive guide to website testing on different devices covers everything you need to know.