BIRT's relationship to the "Datatools project"

[patric-r]

BIRT uses many classes from the datatools project, e.g. for the XML data sources (e.g. org.eclipse.datatools.enablement.oda.xml).

We're facing many issues with XML data sources (mainly performance issues)
E.g. for each data set access from a XML data source, the xml will be parsed again and again (using SAX).
For some input xmls with only a few hundred kilobytes, PDF rendering takes more than 30 seconds and >= 90% of the time is spent in the datatools project and profilers show that same xml is parsed for more than 200 times in our use case/report.

Questions:

• Is there a github project available for datatools?
• Is it actively maintained?
• I saw that many of the classes have been written by Actuate. Most probably, datatools is primarily consumed by birt.
  Wouldn't it make sense to integrate the datatools project into BIRT as well?

[wimjongman]

I think you can lookup the answers to these questions yourself here: https://www.eclipse.org/projects/

[patric-r]

Maybe you can give me a hint how to find the answers there, especially for the most important question, the 3rd one?

I only find the git repo for the project - but this was not a question.
I don't think there's a github project for it but maybe I missed something (or it might be a private repo) that's why I asked.

[wimjongman]

The project page is the central place for every project. From there you can see how many committers there are (2), that the project source is not on Github and that the project is minimally maintained (a couple of commits to get the build working).

I don't think we can include datatools since many projects might rely on them.

The parsing issue you present looks like sloppy programming. I don't think embedding datatools will fix that.

I agree that we should be concerned by the fact that datatools is not very active.

[patric-r]

Embedding will not fix the problem, of course.
But:
We consider fixing the most important performance issues but if it is not part of the BIRT project and I have to use eclipse bugzilla, I fear that our effort might be worthless or not productive (as there are no PRs etc).
Last time, it took 6 years that my patch was finally integrated into datatools (https://bugs.eclipse.org/bugs/show_bug.cgi?id=412269)
Additionally, I don't know how to set up a development environment for datatools...

Alternatively, I can image about an alternative XML Data source parsing mode (maybe we should introduce a checkbox on the data source) which will not use datatools at all, e.g. by using DOM and reading everything (one-time) into memory.
Especially for small xmls and many data set accesses, this would improve performance by magnitudes.
But I'm not sure how this fits to the BIRT architecture (should I create new BIRT subprojects for this? Or even a new xml data source? I don't know how to create BIRT subprojects...).

[wimjongman]

Any patches that need to go into datatools to make BIRT faster should not wait. If the project does not respond, we should go to EMO and complain. I can imagine that you become a datatools committer to represent BIRT.

As for this specific issue, I think it is a BIRT problem, not a datatools problem. Can you pinpoint where it happens in the source?

[patric-r]

Profiling results (this represents a single PDF rendering, containing a single xml data source, using a ~300 kb xml). This takes 41 seconds.

We have several data sets (~10) connected to the xml data source and using xml filter clauses (effectively xpath) within iterable birt structures like Lists.
It looks like BIRT (or data tools) is caching nothing regarding the xml, causing the xml parsed over and over again (including all sub-structures required for BIRT, like XPath objects etc.).

Invocation counts for the central data tools classes for xml parsing for my profiled report generation:

Another issue is that datatools seems to not use the JAXP classes for XML operations (e.g. have a look at org.eclipse.datatools.enablement.oda.xml.util.ChildrenAllowedTreeNode.addChild(String)) but have a proprietary implementation with a limited feature set and many quirks.

[claesrosell]

It actually looks like there will be a change of management in datatools soon. https://www.eclipse.org/lists/dtp-dev/2021/Nov/index.html

The project seems to have both gerrit and ci enabled, with some 230 unit tests, so if you file bugs and push PRs you might have a good chance to get them reviewed and applied.

[wimjongman]

I also noticed that the WWD (Wild Web Developer) project has XML processing features as well.

[patric-r]

Thanks for the update.

Is there a way to import a datatools working copy into the BIRT development environment in order to locally modify datatools but to be able to test it with BIRT as well?

[ruspl-afed]

@patric-r I think you can try to import any bundle from target platform to your BIRT workspace via
"Import" -> "Import Plug-ins and Fragments" (with sources)

[wimjongman]

Use "project from repository". This way you can submit the patches that you make.

[ruspl-afed]

Yes, true.
When I was thinking what to suggest to @patric-r - I thought that finding the right project in the right repository could be tricky itself.

[wimjongman]

I understand. Information about the datatools repo is here:

https://projects.eclipse.org/projects/tools.datatools/developer

[qs-wang]

Just try to add some back ground here. Actuated was working with DTP team together by providing some extensible framework for BIRT to leverage. If @patric-r you'd like to fix anything but is worry about if DTP is going to adopt it, you could try to implement your own data source. This will be only depending on a small piece of DTP which means you can contribute your code either in your won github repo, or as a sub project here in BIRT.

You may want to take a look at org.eclipse.birt.data.oda.mongodb as an example.