Automatic reporting

[sebhoerl]

It would be nice to have an automatic and visual report on the pipeline results. This could just be an HTML document with plots on the outputs and the comparison between the generated population and reference data. This will also help to analyze potential changes in the output data when new components are introduced.

[LeonardoLuquezi]

Hello @sebhoerl ! I was working on this issu with @Nitnelav : We propose a python treatment that feeds a Jinja HTML template with the sythesis data frames from the analysis pipeline section. In addition, to have fancy graphs, we use ECharts, an Open Source JavaScript Visualization Library.

What do you think about this approach ?

Note : I'm coding a beta version with standards analysis found in the bibliography

[sebhoerl]

Really nice! I would have used Vega but just because I'm more familiar with it, but I didn't know ECharts and it looks really nice on first sight! I saw it can also visualize maps, which will be useful. The first steps that I made (also based on Jinja) were basically simply showing the regions / departments / municipalities of the generated area to have an idea of the generated data on top of the report (because lately we were applying the pipeline to more and smaller use cases). So it seems this should also be straight-forward with ECharts.

So this is really nice and very much in line with what I have in mind, thanks for pushing this forward! :)

One big question is what we want to compare in the report: For instance, in the plot above, is this the INSEE aggregated census data (like from the aggregated Excel sheets) vs. the population, or already some census aggregates that have been cleaned and generated by the pipeline from the raw data? It might make sense (if it is available) to compare in every step with data that is as close as possible to the most "official" source.

[sebhoerl]

Hi @LeonardoLuquezi I just uploaded the code that I was working on once in this branch:
https://github.com/eqasim-org/ile-de-france/tree/reporting-draft

The idea was to have a reporting package with one stage that creates the final report, and individual stages that prepare the data to be visualized. This structured data could then later also be used to create visualizations for a paper, aside from the standard automatic report.

This was the basic idea, but I'm open to any other set-up. Just wanted to upload it so it doesn't get lost in my local repository :)

[LeonardoLuquezi]

Awesome, a work in progress branch is good idea @sebhoerl ! Funny, we thought of the same set-up for the reporting section. I'm going to check what you've already done to find out how I can contribute. Thank you !