Categories
html html-parsing php xml xml-parsing

How do you parse and process HTML/XML in PHP?

2257

How can one parse HTML/XML and extract information from it?

0

    1976

    Native XML Extensions

    I prefer using one of the native XML extensions since they come bundled with PHP, are usually faster than all the 3rd party libs and give me all the control I need over the markup.

    DOM

    The DOM extension allows you to operate on XML documents through the DOM API with PHP 5. It is an implementation of the W3C’s Document Object Model Core Level 3, a platform- and language-neutral interface that allows programs and scripts to dynamically access and update the content, structure and style of documents.

    DOM is capable of parsing and modifying real world (broken) HTML and it can do XPath queries. It is based on libxml.

    It takes some time to get productive with DOM, but that time is well worth it IMO. Since DOM is a language-agnostic interface, you’ll find implementations in many languages, so if you need to change your programming language, chances are you will already know how to use that language’s DOM API then.

    How to use the DOM extension has been covered extensively on StackOverflow, so if you choose to use it, you can be sure most of the issues you run into can be solved by searching/browsing Stack Overflow.

    A basic usage example and a general conceptual overview are available in other answers.

    XMLReader

    The XMLReader extension is an XML pull parser. The reader acts as a cursor going forward on the document stream and stopping at each node on the way.

    XMLReader, like DOM, is based on libxml. I am not aware of how to trigger the HTML Parser Module, so chances are using XMLReader for parsing broken HTML might be less robust than using DOM where you can explicitly tell it to use libxml’s HTML Parser Module.

    A basic usage example is available in another answer.

    XML Parser

    This extension lets you create XML parsers and then define handlers for different XML events. Each XML parser also has a few parameters you can adjust.

    The XML Parser library is also based on libxml, and implements a SAX style XML push parser. It may be a better choice for memory management than DOM or SimpleXML, but will be more difficult to work with than the pull parser implemented by XMLReader.

    SimpleXml

    The SimpleXML extension provides a very simple and easily usable toolset to convert XML to an object that can be processed with normal property selectors and array iterators.

    SimpleXML is an option when you know the HTML is valid XHTML. If you need to parse broken HTML, don’t even consider SimpleXml because it will choke.

    A basic usage example is available, and there are lots of additional examples in the PHP Manual.


    3rd Party Libraries (libxml based)

    If you prefer to use a 3rd-party lib, I’d suggest using a lib that actually uses DOM/libxml underneath instead of string parsing.

    FluentDom

    FluentDOM provides a jQuery-like fluent XML interface for the DOMDocument in PHP. Selectors are written in XPath or CSS (using a CSS to XPath converter). Current versions extend the DOM implementing standard interfaces and add features from the DOM Living Standard. FluentDOM can load formats like JSON, CSV, JsonML, RabbitFish and others. Can be installed via Composer.

    HtmlPageDom

    Wa72\HtmlPageDom is a PHP library for easy manipulation of HTML
    documents using DOM. It requires DomCrawler from Symfony2
    components
    for traversing
    the DOM tree and extends it by adding methods for manipulating the
    DOM tree of HTML documents.

    phpQuery

    phpQuery is a server-side, chainable, CSS3 selector driven Document Object Model (DOM) API based on jQuery JavaScript Library.
    The library is written in PHP5 and provides additional Command Line Interface (CLI).

    This is described as “abandonware and buggy: use at your own risk” but does appear to be minimally maintained.

    laminas-dom

    The Laminas\Dom component (formerly Zend_DOM) provides tools for working with DOM documents and structures. Currently, we offer Laminas\Dom\Query, which provides a unified interface for querying DOM documents utilizing both XPath and CSS selectors.

    This package is considered feature-complete, and is now in security-only maintenance mode.

    fDOMDocument

    fDOMDocument extends the standard DOM to use exceptions at all occasions of errors instead of PHP warnings or notices. They also add various custom methods and shortcuts for convenience and to simplify the usage of DOM.

    sabre/xml

    sabre/xml is a library that wraps and extends the XMLReader and XMLWriter classes to create a simple “xml to object/array” mapping system and design pattern. Writing and reading XML is single-pass and can therefore be fast and require low memory on large xml files.

    FluidXML

    FluidXML is a PHP library for manipulating XML with a concise and fluent API.
    It leverages XPath and the fluent programming pattern to be fun and effective.


    3rd-Party (not libxml-based)

    The benefit of building upon DOM/libxml is that you get good performance out of the box because you are based on a native extension. However, not all 3rd-party libs go down this route. Some of them listed below

    PHP Simple HTML DOM Parser

    • An HTML DOM parser written in PHP5+ lets you manipulate HTML in a very easy way!
    • Require PHP 5+.
    • Supports invalid HTML.
    • Find tags on an HTML page with selectors just like jQuery.
    • Extract contents from HTML in a single line.

    I generally do not recommend this parser. The codebase is horrible and the parser itself is rather slow and memory hungry. Not all jQuery Selectors (such as child selectors) are possible. Any of the libxml based libraries should outperform this easily.

    PHP Html Parser

    PHPHtmlParser is a simple, flexible, html parser which allows you to select tags using any css selector, like jQuery. The goal is to assiste in the development of tools which require a quick, easy way to scrape html, whether it’s valid or not! This project was original supported by sunra/php-simple-html-dom-parser but the support seems to have stopped so this project is my adaptation of his previous work.

    Again, I would not recommend this parser. It is rather slow with high CPU usage. There is also no function to clear memory of created DOM objects. These problems scale particularly with nested loops. The documentation itself is inaccurate and misspelled, with no responses to fixes since 14 Apr 16.


    HTML 5

    You can use the above for parsing HTML5, but there can be quirks due to the markup HTML5 allows. So for HTML5 you may want to consider using a dedicated parser. Note that these are written in PHP, so suffer from slower performance and increased memory usage compared to a compiled extension in a lower-level language.

    HTML5DomDocument

    HTML5DOMDocument extends the native DOMDocument library. It fixes some bugs and adds some new functionality.

    • Preserves html entities (DOMDocument does not)
    • Preserves void tags (DOMDocument does not)
    • Allows inserting HTML code that moves the correct parts to their proper places (head elements are inserted in the head, body elements in the body)
    • Allows querying the DOM with CSS selectors (currently available: *, tagname, tagname#id, #id, tagname.classname, .classname, tagname.classname.classname2, .classname.classname2, tagname[attribute-selector], [attribute-selector], div, p, div p, div > p, div + p, and p ~ ul.)
    • Adds support for element->classList.
    • Adds support for element->innerHTML.
    • Adds support for element->outerHTML.

    HTML5

    HTML5 is a standards-compliant HTML5 parser and writer written entirely in PHP. It is stable and used in many production websites, and has well over five million downloads.

    HTML5 provides the following features.

    • An HTML5 serializer
    • Support for PHP namespaces
    • Composer support
    • Event-based (SAX-like) parser
    • A DOM tree builder
    • Interoperability with QueryPath
    • Runs on PHP 5.3.0 or newer

    Regular Expressions

    Last and least recommended, you can extract data from HTML with regular expressions. In general using Regular Expressions on HTML is discouraged.

    Most of the snippets you will find on the web to match markup are brittle. In most cases they are only working for a very particular piece of HTML. Tiny markup changes, like adding whitespace somewhere, or adding, or changing attributes in a tag, can make the RegEx fails when it’s not properly written. You should know what you are doing before using RegEx on HTML.

    HTML parsers already know the syntactical rules of HTML. Regular expressions have to be taught for each new RegEx you write. RegEx are fine in some cases, but it really depends on your use-case.

    You can write more reliable parsers, but writing a complete and reliable custom parser with regular expressions is a waste of time when the aforementioned libraries already exist and do a much better job on this.

    Also see Parsing Html The Cthulhu Way


    Books

    If you want to spend some money, have a look at

    I am not affiliated with PHP Architect or the authors.

    15

    • 11

      @Naveed that depends on your needs. I have no need for CSS Selector queries, which is why I use DOM with XPath exclusively. phpQuery aims to be a jQuery port. Zend_Dom is lightweight. You really have to check them out to see which one you like best.

      – Gordon

      Aug 26, 2010 at 17:38

    • 2

      @Ms2ger Mostly, but not completely. Like already pointed out above, you can use the libxml based parsers but there is special cases where those will choke. If you need maximum compatibility you are better off with a dedicated parser. I prefer to keep the distinction.

      – Gordon

      Jan 19, 2011 at 12:12


    • 9

      Your point for not using PHP Simple HTML DOM Parser seems moot.

      – Petah

      Feb 27, 2012 at 4:47

    • 3

      As of Mar 29, 2012, DOM does not support html5, XMLReader does not support HTML and last commit on html5lib for PHP is on Sep 2009. What to use to parse HTML5, HTML4 and XHTML?

      Mar 29, 2012 at 6:19

    • 6

      @Nasha I deliberately excluded the infamous Zalgo rant from the list above because it’s not too helpful on it’s own and lead to quite some cargo cult since it was written. People were slapped down with that link no matter how appropriate a regex would have been as a solution. For a more balanced opinion, please see the link I did include instead and go through the comments at stackoverflow.com/questions/4245008/…

      – Gordon

      Apr 28, 2015 at 12:35


    336

    Try Simple HTML DOM Parser.

    • A HTML DOM parser written in PHP 5+ that lets you manipulate HTML in a very easy way!
    • Require PHP 5+.
    • Supports invalid HTML.
    • Find tags on an HTML page with selectors just like jQuery.
    • Extract contents from HTML in a single line.
    • Download

    Note: as the name suggests, it can be useful for simple tasks. It uses regular expressions instead of an HTML parser, so will be considerably slower for more complex tasks. The bulk of its codebase was written in 2008, with only small improvements made since then. It does not follow modern PHP coding standards and would be challenging to incorporate into a modern PSR-compliant project.

    Examples:

    How to get HTML elements:

    // Create DOM from URL or file
    $html = file_get_html('http://www.example.com/');
    
    // Find all images
    foreach($html->find('img') as $element)
           echo $element->src . '<br>';
    
    // Find all links
    foreach($html->find('a') as $element)
           echo $element->href . '<br>';
    

    How to modify HTML elements:

    // Create DOM from string
    $html = str_get_html('<div id="hello">Hello</div><div id="world">World</div>');
    
    $html->find('div', 1)->class="bar";
    
    $html->find('div[id=hello]', 0)->innertext="foo";
    
    echo $html;
    

    Extract content from HTML:

    // Dump contents (without tags) from HTML
    echo file_get_html('http://www.google.com/')->plaintext;
    

    Scraping Slashdot:

    // Create DOM from URL
    $html = file_get_html('http://slashdot.org/');
    
    // Find all article blocks
    foreach($html->find('div.article') as $article) {
        $item['title']     = $article->find('div.title', 0)->plaintext;
        $item['intro']    = $article->find('div.intro', 0)->plaintext;
        $item['details'] = $article->find('div.details', 0)->plaintext;
        $articles[] = $item;
    }
    
    print_r($articles);
    

    6

    • 9

      Well firstly there’s things I need to prepare for such as bad DOM’s, Invlid code, also js analysing against DNSBL engine, this will also be used to look out for malicious sites / content, also the as i have built my site around a framework i have built it needs to be clean, readable, and well structured. SimpleDim is great but the code is slightly messy

      Aug 26, 2010 at 17:35


    • 9

      @Robert you might also want to check out htmlpurifier.org for the security related things.

      – Gordon

      Aug 31, 2010 at 7:40

    • 3

      He’s got one valid point: simpleHTMLDOM is hard to extend, unless you use decorator pattern, which I find unwieldy. I’ve found myself shudder just making changes to the underlying class(es) themselves.

      – Erik

      Sep 17, 2010 at 21:46

    • 1

      What I did was run my html through tidy before sending it to SimpleDOM.

      – MB34

      Apr 23, 2012 at 14:14

    • 1

      I’m using this currently, running it as part of a project to process a few hundred urls. It’s becoming very slow and regular timeouts persist. It is a great beginners script and intuitively simple to learn, but just too basic for more advanced projects.

      Apr 7, 2016 at 14:53

    247

    Just use DOMDocument->loadHTML() and be done with it. libxml’s HTML parsing algorithm is quite good and fast, and contrary to popular belief, does not choke on malformed HTML.

    8

    • 20

      True. And it works with PHP’s built-in XPath and XSLTProcessor classes, which are great for extracting content.

      – Kornel

      Nov 27, 2008 at 13:28

    • 8

      For really mangled HTML, you can always run it through htmltidy before handing it off to DOM. Whenever I need to scrape data from HTML, I always use DOM, or at least simplexml.

      Oct 13, 2009 at 0:41

    • 10

      Another thing with loading malformed HTML i that it might be wise to call libxml_use_internal_errors(true) to prevent warnings that will stop parsing.

      – Husky

      May 24, 2010 at 17:51

    • 6

      I have used DOMDocument to parse about 1000 html sources (in various languages encoded with different charsets) without any issues. You might run into encoding issues with this, but they aren’t insurmountable. You need to know 3 things: 1) loadHTML uses meta tag’s charset to determine encoding 2) #2 can lead to incorrect encoding detection if the html content doesn’t include this information 3) bad UTF-8 characters can trip the parser. In such cases, use a combination of mb_detect_encoding() and Simplepie RSS Parser’s encoding / converting / stripping bad UTF-8 characters code for workarounds.

      – Zero

      Sep 19, 2010 at 6:58


    • 1

      DOM does actually support XPath, take a look at DOMXPath.

      Jan 30, 2012 at 2:59