> ## Documentation Index
> Fetch the complete documentation index at: https://extended-openai-conversation.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Scrape Functions

> Extract data from web pages

Scrape functions extract data from HTML pages using CSS selectors, similar to Home Assistant's [scrape integration](https://www.home-assistant.io/integrations/scrape/). They're useful for websites without APIs.

## Configuration

```yaml theme={null}
function:
  type: scrape
  resource: https://example.com
  value_template: "{{ value }}"
  sensor:
    - name: sensor_name
      select: ".css-selector"
      value_template: "{{ value }}"
```

<ParamField path="resource" type="string" required>
  The URL of the web page to scrape
</ParamField>

<ParamField path="sensor" type="array" required>
  List of data points to extract from the page
</ParamField>

<ParamField path="value_template" type="string" required>
  Template that combines scraped sensor values into the function result
</ParamField>

## Sensor Configuration

Each sensor extracts a piece of data from the page:

<ParamField path="name" type="string" required>
  Variable name to use in the main `value_template`
</ParamField>

<ParamField path="select" type="string" required>
  CSS selector to find the element
</ParamField>

<ParamField path="value_template" type="string">
  Template to process the selected element's text
</ParamField>

<ParamField path="attribute" type="string">
  Extract an HTML attribute instead of text (e.g., `href`, `src`)
</ParamField>

## CSS Selectors

<AccordionGroup>
  <Accordion title="Basic Selectors">
    ```css theme={null}
    .classname        /* Select by class */
    #idname          /* Select by ID */
    tagname          /* Select by tag */
    [attribute]      /* Select by attribute */
    ```
  </Accordion>

  <Accordion title="Combinators">
    ```css theme={null}
    parent child           /* Descendant */
    parent > child         /* Direct child */
    element + sibling      /* Adjacent sibling */
    element ~ sibling      /* General sibling */
    ```
  </Accordion>

  <Accordion title="Attribute Selectors">
    ```css theme={null}
    [href^="https"]        /* Starts with */
    [href$=".pdf"]         /* Ends with */
    [href*="example"]      /* Contains */
    [data-id="123"]        /* Exact match */
    ```
  </Accordion>

  <Accordion title="Pseudo-classes">
    ```css theme={null}
    :first-child           /* First child */
    :last-child            /* Last child */
    :nth-child(2)          /* Nth child */
    :not(.excluded)        /* Negation */
    ```
  </Accordion>
</AccordionGroup>

## Advanced Features

<AccordionGroup>
  <Accordion title="Extract Attributes">
    Get HTML attributes instead of text:

    ```yaml theme={null}
    sensor:
      - name: image_url
        select: "img.product-image"
        attribute: src
      - name: link
        select: "a.read-more"
        attribute: href
    ```
  </Accordion>

  <Accordion title="Multiple Elements">
    Extract all matching elements:

    ```yaml theme={null}
    sensor:
      - name: all_prices
        select: ".price"
        index: all  # Returns a list
    ```
  </Accordion>

  <Accordion title="Headers">
    Add custom headers (e.g., for authentication):

    ```yaml theme={null}
    function:
      type: scrape
      resource: https://example.com
      headers:
        User-Agent: "Mozilla/5.0"
        Cookie: "session=abc123"
      sensor:
        - name: data
          select: ".content"
    ```
  </Accordion>
</AccordionGroup>

## Use Cases

<CardGroup cols={2}>
  <Card title="Version Tracking" icon="code-branch">
    Monitor software versions and releases
  </Card>

  <Card title="Price Monitoring" icon="dollar-sign">
    Track product prices and availability
  </Card>

  <Card title="News Aggregation" icon="newspaper">
    Extract headlines and articles
  </Card>

  <Card title="Status Pages" icon="circle-dot">
    Monitor service status pages
  </Card>
</CardGroup>

## Best Practices

<Steps>
  <Step title="Inspect the page">
    Use browser DevTools to find the right CSS selectors. Right-click an element and select "Inspect" to see its HTML structure.
  </Step>

  <Step title="Handle missing elements">
    Pages may change. Use safe filters and defaults:

    ```jinja2 theme={null}
    {{ value | default('Not found') }}
    ```
  </Step>

  <Step title="Respect robots.txt">
    Check the website's `robots.txt` to ensure scraping is allowed. Add appropriate delays for rate limiting.
  </Step>

  <Step title="Add User-Agent">
    Some sites block requests without a User-Agent header:

    ```yaml theme={null}
    headers:
      User-Agent: "Home Assistant Extended OpenAI Conversation"
    ```
  </Step>
</Steps>

## Debugging

Test CSS selectors in browser console:

```javascript theme={null}
document.querySelector('.css-selector')
document.querySelectorAll('.css-selector')
```

Enable debug logging:

```yaml theme={null}
logger:
  logs:
    custom_components.extended_openai_conversation: debug
```

## Examples

### Get Home Assistant Version

Scrape the current version from home-assistant.io:

```yaml theme={null}
- spec:
    name: get_ha_version
    description: Use this function to get Home Assistant version
    parameters:
      type: object
      properties:
        dummy:
          type: string
          description: Not used (placeholder)
  function:
    type: scrape
    resource: https://www.home-assistant.io
    value_template: "version: {{version}}, release_date: {{release_date}}"
    sensor:
      - name: version
        select: ".current-version h1"
        value_template: '{{ value.split(":")[1] }}'
      - name: release_date
        select: ".release-date"
        value_template: '{{ value.lower() }}'
```

<Frame>
  <img width="300" alt="Scrape Example" src="https://github.com/jekalmin/extended_openai_conversation/assets/2917984/e640c3f3-8d68-486b-818e-bd81bf71c2f7" />
</Frame>

### Scrape Product Price

```yaml theme={null}
- spec:
    name: get_product_price
    description: Get current price of a product
    parameters:
      type: object
      properties:
        url:
          type: string
          description: Product page URL
      required:
      - url
  function:
    type: scrape
    resource: "{{ url }}"
    value_template: "Price: {{ price }}, In Stock: {{ stock }}"
    sensor:
      - name: price
        select: ".price-current"
        value_template: '{{ value | replace("$", "") | float }}'
      - name: stock
        select: ".stock-status"
        value_template: '{{ "yes" if "in stock" in value.lower() else "no" }}'
```

### Extract Multiple Items

```yaml theme={null}
- spec:
    name: get_news_headlines
    description: Get latest news headlines
    parameters:
      type: object
      properties:
        dummy:
          type: string
  function:
    type: scrape
    resource: https://news.ycombinator.com
    value_template: >-
      Top Stories:
      {% for i in range(1, 6) %}
      {{ i }}. {{ headlines[i-1] }}
      {% endfor %}
    sensor:
      - name: headlines
        select: ".titleline > a"
        value_template: '{{ value }}'
        index: all
```

## Comparison: Scrape vs REST

| Feature       | Scrape                | REST                |
| ------------- | --------------------- | ------------------- |
| Data format   | HTML                  | JSON/XML            |
| Requires API  | No                    | Yes                 |
| Selector type | CSS                   | JSON path           |
| Best for      | Public websites       | APIs with keys      |
| Maintenance   | Higher (HTML changes) | Lower (stable APIs) |

## Next Steps

<CardGroup cols={3}>
  <Card title="REST Functions" icon="globe" href="/functions/rest">
    Use APIs when available
  </Card>

  <Card title="Composite Functions" icon="layer-group" href="/functions/composite">
    Combine scraping with processing
  </Card>

  <Card title="Scrape Integration" icon="book" href="https://www.home-assistant.io/integrations/scrape/">
    HA scrape integration docs
  </Card>
</CardGroup>