Automatic report testing

[doortokaos]

Hi,

since nowadays, BIRT produces new releases quite regularly I want to keep up with the releases and integrate the new versions in my product regularly.
In my product there are some integrated reports that can be used in different output formats (PDF, DOCX, XLSX). The product itself is written in JAVA, and we already have JUnit-tests for our code.

I'm trying to figure out a good way to automatically test the reports for correct generation and errorless content after an update of the report engine. Manual testing can do it's part, but I want to automate the process, so save time and manual effort if possible.

I appreciate any ideas on how to achieve automatic report testing.

Best regards,
doortokaos

[hvbtup]

Good question. I think the best one can do for testing regressions in the PDF output is to create the reports in such a way that their output is independent on the of current date/time and then use Apache PDFBox or similar tools to generate bitmap images from the PDF and then XOR the pixels for each colour channel with the "golden" image, allowing only a very small fraction of the XOR values to be non-zero.

[doortokaos]

Hm yes, that would work for PDF, but I would get no hint, if other formats are working or the layout has changed.

The emitters tend to render things quite differently, even between PDF and DOCX the documents won't look exactly the same (which is ok, I think, but therefore I cannot assume everything will be ok in DOCX as well, when PDF is fine).

For the content of the report you could create the report as HTML and try to analyse the content of the given HTML file (should be doable with "intelligent" XPath queries), but that won't help for layout as well.

I'll leave this topic open, perhaps there are other ideas.