How to Wait for Page to Fully Load in Playwright

If you‘ve ever written a web scraper or automated testing script, you know how critical it is to ensure the page has fully loaded before attempting to interact with it or extract data. Failing to properly wait is a common cause of flaky, unreliable scripts that work only intermittently.

As an expert in web scraping and automation, I frequently see scripts fail because they don‘t account for the complex, asynchronous nature of modern web pages. It‘s not enough to just navigate to a URL and immediately start clicking and scraping – you need to intelligently wait for the right network requests to finish and elements to appear.

In this in-depth guide, I‘ll walk through how to use Playwright to properly wait for pages to fully load. I‘ll clarify what "fully loaded" really means, demonstrate the key waiting methods with practical examples, and share some pro tips to make your scripts faster and more resilient.

By the end of this guide, you‘ll be able to confidently handle even the most dynamic, AJAX-heavy pages with Playwright.

What Does "Fully Loaded" Mean?

First, let‘s establish a shared understanding of what it means for a web page to be considered "fully loaded".

Modern websites are highly complex and resource-intensive. The initial HTML document is just the tip of the iceberg. After parsing the HTML, the browser must fetch and parse external stylesheets, execute synchronous JavaScript, perform layout calculations, paint pixels to the screen, and often make dozens of asynchronous requests for more data.

Consider these eye-opening statistics:

• The median webpage is now 2 MB in size (source)
• The median page makes 73 requests (source)
• 10% of pages take over 8 seconds to reach full interactivity (source)

To further complicate matters, many modern sites are built as single-page applications (SPAs) where the initial page load is just a bare-bones HTML shell, and all meaningful content is fetched dynamically via AJAX. Techniques like lazy loading and infinite scroll mean that a page may never reach a definitive "finished" state.

So in short, a fully loaded page in the context of browser automation is one where:

1. The main HTML document has been parsed and its DOM is ready
2. All subresources like stylesheets, scripts, and images are fetched & processed
3. Any dynamic content triggered by the initial page load is fetched & rendered

With this definition in mind, let‘s look at how Playwright helps us determine when a page is ready.

Playwright‘s Default Page Load Waiting Behavior

By default, when you use the page.goto() method to navigate to a URL, Playwright will wait for the initial page load to complete before moving on to the next line of code.

Under the hood, Playwright is listening for the load event to fire in the page. This event is triggered when the whole page, including all dependent resources such as stylesheets and images, has finished loading (source).

Here‘s a minimal example demonstrating this default behavior:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:  
    browser = p.chromium.launch()
    page = browser.new_page()

    page.goto(‘https://example.com‘) 

    # Playwright will wait for page to fire the load event
    print(page.title())

    browser.close()

This default waiting is a good start, but has some key limitations:

• It doesn‘t wait for asynchronous requests triggered after the load event
• It doesn‘t ensure that page has actually rendered meaningful content
• There‘s no way to adjust the waiting criteria for specific pages

To reliably scrape pages with dynamic content, we need more precise control over when Playwright considers a page "ready". Fortunately, Playwright provides several methods to wait for specific elements and network events.

Waiting for Elements to Appear

The most common technique to determine that a page is fully loaded is to wait for specific elements to be present in the DOM. These elements serve as landmarks, indicating that the prerequisite content and resources have finished loading.

wait_for_selector

The page.wait_for_selector() method is the Swiss Army knife of element waiting. It accepts a CSS selector or other locator, and blocks until an element matching that selector appears in the page, or until the maximum wait time is reached.

page.goto(‘https://example.com‘)

# Wait up to 10 seconds for an element matching this selector 
page.wait_for_selector(‘#search-results‘, timeout=10000)

Under the hood, Playwright is polling the page at a regular interval, running document.querySelector for the given selector. As soon as the selector matches at least one element, or the timeout is reached, the method returns.

You can use any valid CSS selector, as well as other selector engines like XPath, if you first register them with playwright.selectors.register().

wait_for_load_state

For finer-grained control over what load event to wait for, we can use page.wait_for_load_state(). It accepts one of three states:

• ‘load‘: wait for the full page load event (default behavior)
• ‘domcontentloaded‘: wait for the DOMContentLoaded event
• ‘networkidle‘: wait for the page to have no more than 0 network requests for at least 500 ms

The ‘domcontentloaded‘ option is useful when you need to access the page sooner than the full load event, but still want to ensure the DOM is interactive. For example:

page.goto(‘https://example.com‘)

page.wait_for_load_state(‘domcontentloaded‘)

# Start interacting with the DOM before external resources are loaded
form = page.query_selector(‘#search-form‘) 

We‘ll discuss the ‘networkidle‘ option in more detail in the next section.

Here‘s a more realistic example combining element and load state waiting:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()

    page.goto(‘https://www.amazon.com/s?k=laptop‘)

    # Wait for the search results container to appear
    page.wait_for_selector(‘.s-result-list‘)

    # Wait for any lazy loaded images to finish
    page.wait_for_load_state(‘networkidle‘)

    # Extract the product data
    products = page.query_selector_all(‘.s-result-item‘)

    for product in products:
        name = product.query_selector(‘h2‘).text_content()
        price = product.query_selector(‘.a-price‘).text_content()
        print(f‘{name}: {price}‘)

    browser.close()

In this snippet, we first wait for the .s-result-list element to appear, indicating that the initial search results have loaded. We then wait for network idleness to ensure any lazy loaded images have finished before scraping the product data.

Waiting for Network Requests to Complete

The ‘networkidle‘ load state option is invaluable for waiting on asynchronous requests to finish after the initial page load. When would you need this?

Consider a search results page where the initial set of results loads quickly, but then more results are fetched and appended to the page as you scroll. The scroll event triggers an XHR request for more data, which is rendered after it arrives. If you try to scrape the full result set immediately after the first load, you‘ll miss the later results.

With page.wait_for_load_state(‘networkidle‘), Playwright will wait until there have been no more than 0 active network requests for at least 500 ms. In other words, it waits for a half second of network inactivity after the page first loads.

You can adjust the network idleness criteria with the networkIdleTimeout and networkIdleInflight options:

# Wait for no more than 2 active connections for at least 1000ms
page.wait_for_load_state(‘networkidle‘, networkIdleTimeout=1000, networkIdleInflight=2)

Another common pattern in modern web apps is to fetch data in the background and then update the page URL to reflect the new application state (sometimes called the AJAX URL). For example, a mapping app might update the URL with the coordinates and zoom level after you interact with the map.

To wait for a URL update like this, we can use page.wait_for_url():

# Wait for the URL to include #search_results
page.wait_for_url(re.compile(r‘#search_results‘))

We can pass a string, regular expression, or predicate function to specify the URL we‘re waiting for.

Here‘s an example combining network and URL waiting:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()

    page.goto(‘https://www.redfin.com/city/30818/CA/San-Francisco‘)

    # Wait for the initial map to load
    page.wait_for_selector(‘#MapContainer‘)

    # Interact with the map
    page.mouse.wheel(0, 2000) 
    page.mouse.click(200, 200)

    # Wait for the URL to reflect the updated map state  
    page.wait_for_url(re.compile(r‘/CA/San-Francisco/coord‘))

    # Wait for any new background requests to finish
    page.wait_for_load_state(‘networkidle‘)  

    # Extract the updated map data
    properties = page.query_selector_all(‘.HomeViews‘)
    for prop in properties:
        print(prop.query_selector(‘.streetAddress‘).text_content())

    browser.close()

This script interacts with a map to update the visible location, then waits for the URL to change and any new map data to load before extracting the visible properties.

Setting Custom Timeouts

By default, most of Playwright‘s waiting methods will wait up to 30 seconds before timing out. In some cases, you may need to increase or decrease this timeout based on the behavior of the site you‘re testing.

To set a custom timeout for a specific action, pass the timeout option in milliseconds:

# Wait up to 60 seconds for the #search-results element
page.wait_for_selector(‘#search-results‘, timeout=60000)

# Wait up to 45 seconds for network requests to finish
page.wait_for_load_state(‘networkidle‘, timeout=45000)

You can also set global timeouts that apply to all actions using the timeout option in your Playwright configuration:

playwright = sync_playwright().start(timeout=5000)

When deciding on timeouts, consider the following:

• What‘s the typical full load time for your page on your test environment? Use your 95th percentile load time as a starting point.
• What‘s the impact of a too-short timeout? Might be better to fail slowly than to have flaky failures.
• Are there certain actions that might reasonably take longer, like a complex search or report generation? Set custom timeouts for these.

In general, start with a longer timeout and then optimize it down as you build confidence in the page‘s performance. It‘s better to have a slightly slower but stable script than one that fails intermittently due to overly aggressive timeouts.

Tips and Best Practices

Here are a few more tips to make your Playwright scripts more reliable and resilient:

1. Identify the minimal set of elements that signal the page is ready for your use case. Don‘t wait for the entire page to finalize if you only need a certain piece of it. Waiting more than necessary slows down your scripts.

2. Use specific, unique selectors for elements you wait for. Waiting for generic elements like div or span can cause false positives if another matching element appears before your target element. Use IDs, data attributes, or unique class combinations to make the selector as specific as possible.

3. Watch for page load errors. If a page fails to load due to a network error or unhandled exception, Playwright will throw an exception. Catch and handle these exceptions gracefully in your script, perhaps by retrying the operation or logging the error for later investigation.

4. Remember that Playwright auto-waits for elements before interacting with them. When you call methods like click() or fill(), Playwright will automatically wait for the target element to be visible and actionable. You don‘t need to explicitly wait for an element if your next action is to interact with that same element.

5. Test your waiting criteria across different environments and conditions. A selector that works great locally might fail in CI or staging due to different network speeds or dynamic IDs. Run your script in a variety of environments, and consider making waiting criteria configurable for different contexts.

6. Optimize judiciously. Once you have a working, stable script, profile its performance and look for opportunities to optimize, like reducing unnecessary waits or running independent operations concurrently. But avoid premature optimization that sacrifices readability or reliability.

Conclusion

Waiting for web pages to fully load is both a science and an art. By understanding the page loading process, using Playwright‘s waiting APIs effectively, and following best practices, you can create robust, resilient scripts that work reliably across a wide variety of pages and environments.

Some key techniques to remember:

• Wait for specific elements that signal the page is ready for your use case
• Wait for network idleness if you need data from background requests
• Wait for URL changes if the page URL reflects updated state
• Set appropriate timeouts based on the page‘s actual performance
• Handle load errors and test waiting criteria across environments

Proper waiting is critical to creating stable, maintainable scripts. But it‘s just one piece of the puzzle. As you use Playwright more, strive to continually refactor and optimize your scripts to keep them fast, clear, and effective.

I hope this guide has given you a solid foundation on waiting for pages in Playwright. Remember, every site is different, so the best teacher is hands-on experimentation. Try out these techniques on your own pages and see what works best for your specific needs.

Happy automated testing!