Random errors when append with another Documents html()

[adrian-dybwad]

We are using jsoup in an HttpServlet and replacing a div with the body from a file we get from the filesystem.

All works well, until we get random errors that seem to be related to multiple simultaneous requests on the servlet.

We use the following code to replace the content of a div of a Document object with the body of another Document object:

public class TemplateMerger {

    private Document mergedRemoteTemplate;

    //remoteTemplate is a Document object that is stored as a static variable in the calling class
    public TemplateMerger(final Document remoteTemplate) {
        this.mergedRemoteTemplate = remoteTemplate.clone();
    }

    //with is an html file read from the local file system by the calling class
    public Document mergeWith(final Document with) {
        //mergedRemoteTemplate has a div with id "MainContent"
        Element remoteTemplateMainContent = mergedRemoteTemplate.getElementById("MainContent");
        remoteTemplateMainContent.append(with.body().html());
        return mergedRemoteTemplate;
    }

}

When doing remoteTemplateMainContent.append(with.body().html()), there are about 30 different errors that are randomly thrown (by the jsoup parser) with some examples:
Cannot invoke "java.util.ArrayList.isEmpty()" because "this.stack" is null
java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -7 is negative
org.jsoup.helper.ValidationException: String must not be empty
org.jsoup.helper.ValidationException: Object must not be null
java.lang.NullPointerException: Cannot invoke "String.hashCode()" because "" is null

I believe it is the result of with.body().html() that is being changed by another request thread while the call to remoteTemplateMainContent.append is in progress.

These errors seem to have happened in jsoup version 1.18.1 but only once in a while, but in the latest version (1.18.3) it is happening often. At least we now know where they were coming from but preventing them is now our goal.

What are we doing wrong?

[adrian-dybwad]

I solved the errors by creating a static synchronized helper that is the only place where we call parse or connect functions in Jsoup. A non exhaustive set of calls:

import java.io.File;
import java.io.IOException;
import lombok.extern.slf4j.Slf4j;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;

@Slf4j
public class JsoupParser {
  
  public JsoupParser() {
    //No instance
  }
  
  public static Document parse(final String html) {
    return parse(JsoupParseArguments.builder().html(html).build());
  }
  
  public static Document parse(final String html, final String baseUri) {
    return parse(JsoupParseArguments.builder().html(html).baseUri(baseUri).build());
  }
  
  public static Document parse(final File file, final String charsetName) {
    return parse(JsoupParseArguments.builder().file(file).charsetName(charsetName).build());
  }
  
  public static Document connect(final String url) {
    return parse(JsoupParseArguments.builder().connect(url).build());
  }
  
  public static synchronized Document parse(final JsoupParseArguments args) {
    
    if (args.getConnect() != null) {
      try {
        return Jsoup.connect(args.getConnect()).get();
      } catch (IOException e) {
        log.error("Jsoup unable to connect {}", args.getConnect(), e);
      }
    } else if (args.getHtml() != null && args.getBaseUri() != null) {
      return Jsoup.parse(args.getHtml(), args.getBaseUri());
    } else if (args.getFile() != null && args.getCharsetName() != null) {
      try {
        return Jsoup.parse(args.getFile(), args.getCharsetName());
      } catch (IOException e) {
        log.error("Jsoup unable to open file", e);
      }
    } else if (args.getHtml() != null) {
      //Default return statement will be used
    } else {
      log.error("JsoupParser parse unknown arg combination. Result may be null.");
    }
    return Jsoup.parse(args.getHtml());
    
  }

}

import java.io.File;
import lombok.Builder;
import lombok.Getter;

@Builder
@Getter
public class JsoupParseArguments {
  
  private String connect;
  private String html;
  private String baseUri;
  private File file;
  private String charsetName;

}

Now looking to find all parts of Jsoup that share the same memory and that must be included in this function to make it thread safe.

[jhy]

Hi there,

This was due to the document.clone() not making a clone of the Document's parser. So when multiple threads were concurrently calling doc.append(html), the same parser was getting hit concurrently. As a parser instance is explicitly not thread-safe (because it maintains state of the current parse), this would blow up.

Fixed by #2281 by cloning the Parser when the Document is cloned.

BTW you could probably make your implementation more efficient by changing:

remoteTemplateMainContent.append(with.body().html());

to

remoteTemplateMainContent.appendChild(with.body());

Which will save a HTML serialize and subsequent parse on every request (and also would work around this issue, as the parser is not called).

[adrian-dybwad]

Oh excellent, it was a bug! Thanks for the tip on appendChild.