How to Find Elements by CSS Selector in Puppeteer

As a web developer or QA engineer, you may have heard of Puppeteer – the powerful Node.js library developed by Google that allows you to control a headless Chrome or Chromium browser. Puppeteer is an excellent tool for automating web testing, generating PDFs of web pages, capturing screenshots, and web scraping.

One of the most fundamental tasks when working with Puppeteer is finding elements on a page so you can interact with them – clicking buttons, entering text, extracting data, and so on. While there are several ways to locate elements, using CSS selectors is often the most concise and convenient option.

In this guide, we‘ll take an in-depth look at how to find elements by CSS selector in Puppeteer. Whether you‘re new to Puppeteer or just looking to level up your skills, by the end of this post you‘ll be a pro at using CSS selectors to locate elements on a page.

What are CSS Selectors?

Before we dive into finding elements in Puppeteer, let‘s take a step back and briefly discuss what CSS selectors are. If you‘re already familiar with CSS selectors, feel free to jump ahead to the next section.

CSS (Cascading Style Sheets) is a language used to describe the presentation of a document written in HTML. CSS selectors are patterns used to select and style elements on a web page. They allow you to target elements based on their tag name, class, ID, attribute values, relationship to other elements, and more.

Here are some common examples of CSS selectors:

• Element selector: Selects all elements with the specified tag name
  • Example: p selects all <p> elements
• Class selector: Selects all elements with the specified class
  • Example: .highlight selects all elements with class="highlight"
• ID selector: Selects the element with the specified ID
  • Example: #main selects the element with id="main"
• Attribute selector: Selects elements based on their attribute values
  • Example: [type="submit"] selects all elements with type="submit"
• Descendant combinator: Selects elements that are descendants of another element
  • Example: div p selects all <p> elements inside <div> elements

CSS selectors are very powerful and allow you to target elements with a high degree of specificity. For a complete reference on CSS selectors, check out the MDN documentation.

Finding Elements by CSS Selector

Now that we‘ve covered what CSS selectors are, let‘s look at how to use them to find elements in Puppeteer. Puppeteer provides two main methods for locating elements by CSS selector:

1. page.$(selector) – Returns the first element that matches the specified selector.
2. page.$$(selector) – Returns an array of all elements that match the specified selector.

Let‘s look at each of these in more detail.

Using page.$()

The page.$() method allows you to select a single element on the page by its CSS selector. It returns a Promise that resolves to the first matching ElementHandle, or null if no element is found.

Here‘s the basic syntax:

const element = await page.$(‘CSS_SELECTOR‘);

For example, to find the first <h1> element on a page:

const puppeteer = require(‘puppeteer‘);

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto(‘https://example.com‘);

  const heading = await page.$(‘h1‘);
  console.log(await heading.evaluate(node => node.innerText)); 

  await browser.close();
})();

This code launches a browser instance, navigates to https://example.com, finds the first <h1> element on the page using page.$(), and prints out its text content using the evaluate() method.

Using page.$$()

The page.$$() method allows you to select multiple elements that match the specified CSS selector. It returns a Promise that resolves to an array of ElementHandle objects.

Here‘s the basic syntax:

const elements = await page.$$(‘CSS_SELECTOR‘);

For example, to find all the <p> elements on a page:

const puppeteer = require(‘puppeteer‘);

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();

  await page.goto(‘https://example.com‘);

  const paragraphs = await page.$$(‘p‘);
  console.log(`Found ${paragraphs.length} <p> elements`);

  for (const paragraph of paragraphs) {
    console.log(await paragraph.evaluate(node => node.innerText));
  }

  await browser.close();
})();

This code finds all the <p> elements on the page using page.$$(), prints out the total number of elements found, and then loops through each one to log its text content.

Advanced CSS Selectors

In the examples above, we used simple tag name selectors to find elements. But CSS selectors allow for much more advanced and specific targeting. Here are a few more examples of CSS selectors you can use in Puppeteer:

Combinators

Combinators allow you to select elements based on their relationship to other elements. There are four types of combinators:

• Descendant combinator: Selects elements that are descendants of another element
  • Example: div p selects all <p> elements inside <div> elements
• Child combinator: Selects elements that are direct children of another element
  • Example: ul > li selects all <li> elements that are direct children of a <ul> element
• Adjacent sibling combinator: Selects the first element that is the next sibling of another element
  • Example: h1 + p selects the first <p> element that comes immediately after an <h1> element
• General sibling combinator: Selects all elements that are siblings of another element
  • Example: h1 ~ p selects all <p> elements that are siblings of an <h1> element

await page.$$(‘ul > li‘); // Selects all <li> elements that are direct children of a <ul>

await page.$(‘h1 + p‘); // Selects the first <p> element immediately after an <h1>

Pseudo-classes

Pseudo-classes allow you to select elements based on a certain state or condition. Some common pseudo-classes include:

• :first-child – Selects the first child element
• :last-child – Selects the last child element
• :nth-child(n) – Selects elements based on their position among siblings
• :focus – Selects the element that currently has focus
• :hover – Selects elements when you mouse over them

await page.$(‘p:first-child‘); // Selects the first <p> element

await page.$$(‘li:nth-child(odd)‘); // Selects all odd <li> elements

Attribute Selectors

Attribute selectors allow you to select elements based on their attribute values. You can match attributes in several ways:

• [attribute] – Selects elements with the specified attribute
• [attribute="value"] – Selects elements with the specified attribute value
• [attribute^="value"] – Selects elements whose attribute value starts with the specified value
• [attribute$="value"] – Selects elements whose attribute value ends with the specified value
• [attribute*="value"] – Selects elements whose attribute value contains the specified value

await page.$$(‘[type="submit"]‘); // Selects all elements with type="submit" 

await page.$(‘a[href^="https://"]‘); // Selects <a> elements where the href starts with "https://"

Error Handling

When using page.$() or page.$$() to find elements, it‘s important to handle the case where no elements are found. If no elements match the specified selector, page.$() will return null, while page.$$() will return an empty array.

Here‘s an example of how to handle a null return value from page.$():

const element = await page.$(‘#nonexistent‘);

if (!element) {
  console.log(‘Element not found!‘);
} else {
  console.log(await element.evaluate(node => node.innerText));
}

And here‘s an example of checking for an empty array from page.$$():

const elements = await page.$$(‘.missing‘);

if (elements.length === 0) {
  console.log(‘No elements found.‘);
} else {
  console.log(`Found ${elements.length} elements.`);
}

CSS Selectors vs Other Locators

While CSS selectors are a very powerful and commonly used way to locate elements, they‘re not the only option. Puppeteer also supports other locators like XPath and text matching.

XPath

XPath is a query language used to navigate through elements and attributes in an XML document. It can also be used to locate elements on a webpage. Compared to CSS selectors, XPath expressions tend to be more verbose and less readable. However, XPath does offer some advanced features that CSS selectors lack, like the ability to traverse upwards to parent elements.

To use XPath in Puppeteer, you can use the page.$x() method:

const [element] = await page.$x(‘//p[@class="intro"]‘);

Text Matching

Puppeteer also allows you to locate elements by matching against their text content using the page.evaluateHandle() method and passing in a function that checks the textContent of elements:

const elements = await page.evaluateHandle(() => 
  [...document.querySelectorAll(‘p‘)].filter(
    element => element.textContent.includes(‘search term‘)
  )
);

Text matching can be useful when the text content is known but the element structure or attributes are unpredictable. However, it can also be fragile if the text is prone to changing.

In general, CSS selectors are the preferred method for locating elements due to their simplicity, specificity, and maintainability. XPath and text matching are good to be aware of as alternative options for more advanced use cases.

Conclusion

To recap, finding elements is a critical aspect of automating interactions with web pages using Puppeteer. CSS selectors provide a concise and powerful way to locate one or more elements on a page using attributes like tag name, class, ID, and more.

The page.$() and page.$$() methods in Puppeteer allow you to find one or multiple elements, respectively, using a CSS selector string. It‘s important to gracefully handle cases where no elements are found by checking for null return values or empty arrays.

More advanced CSS selector features like combinators, pseudo-classes, and attribute selectors allow you to get very specific with which elements you target. For alternative approaches, Puppeteer also supports XPath selectors and matching elements by their text content.

I hope this guide has helped you gain a solid understanding of using CSS selectors in Puppeteer to find elements. The concepts covered here are fundamental building blocks for writing effective Puppeteer scripts to automate tasks on the web. As you put this knowledge into practice, you‘ll be able to efficiently locate and interact with elements to suit your specific automation needs.

Happy Puppeteering!