Add runs summary as Markdown table

[mole99]

This PR creates a summary for the simulation runs of a parameter using the ngspice tool.
CACE saves the summary as simulation_summary.md in the parameter directory and prints a rendered version.
The first column contains the run number, the following columns contain the changing conditions, and the last columns contain the results, which may look as follows:

  run        b   corner   temperature    vout   offset_error  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  run_00    25       ss           -40   0.334      3.693e-03  
  run_01   230       ss           -40   3.299          0.101  
  run_02    25       tt           -40   0.335      3.749e-03  
  run_03   230       tt           -40   3.300          0.102  
  run_04    25       ff           -40   0.335      3.809e-03  
  run_05   230       ff           -40   3.300          0.102  
  run_06    25       ss            27   0.334      3.545e-03  
  run_07   230       ss            27   3.092      3.865e-02  
  run_08    25       tt            27   0.334      3.597e-03  
  run_09   230       tt            27   3.028      1.918e-02  
  run_10    25       ff            27   0.334      3.651e-03  
  run_11   230       ff            27   3.020      1.658e-02  
  run_12    25       ss           110   0.333      3.325e-03  
  run_13   230       ss           110   3.009      1.339e-02  
  run_14    25       tt           110   0.333      3.380e-03  
  run_15   230       tt           110   3.007      1.287e-02  
  run_16    25       ff           110   0.334      3.485e-03  
  run_17   230       ff           110   3.003      1.165e-02

Implements part of #122.

Future work: add a column indicating whether a single value passed or failed the spec (✅/❌). This requires CACE to check each result individually instead of forming a max/min over the range of values.

[areda0]

Very nice Leo. this would be very useful. you think we can have the same table in a .xlsx format? having the detailed results in a more structured format with built-in filters would be much more useful and would allow us to easily generate graphs, as we wish, for any set of the results across any set of the conditions which is very useful as a reporting feature. otherwise, we would have to configure the .yaml to generate plots across conditions which can be cumbersome and less versatile compared to the .xlsx.

@RTimothyEdwards , what do you think? I feel like this would render CACE a design-phase debugging tool as well.

[mole99]

Thanks Ahmed. Interesting idea, but I'm not a fan of .xlsx as it is a proprietary file format. Better would be the OpenDocument .ods format, or even a plain .csv file (but then you would need to add the filters yourself).

One problem is to represent the table as a whole, as it has a three-dimensional structure. The results can be a list of values, e.g. for a transient or MC sim. The summary only shows the first three elements and then an ellipsis, as otherwise the table would become too large. This is sufficient to get a quick overview, but does not bloat the table:

MC sim:

  run     temperature                 iterations                            a0  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  run_0           -40   [1.000, 2.000, 3.000, …]   [78.572, 78.818, 76.977, …]  
  run_1            27   [1.000, 2.000, 3.000, …]   [79.416, 79.723, 77.904, …]  
  run_2           130   [1.000, 2.000, 3.000, …]   [77.868, 77.974, 76.636, …]

transient sim:

  run     corner   temperature                               time                                   Vout                       Vinp  
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 
  run_0       ss           -40   [0.000, 5.000e-11, 1.000e-10, …]               [0.248, 0.248, 0.248, …]   [0.000, 0.000, 0.000, …]  
  run_1       tt           -40   [0.000, 5.000e-11, 1.000e-10, …]               [0.231, 0.231, 0.231, …]   [0.000, 0.000, 0.000, …]  
  run_2       ff           -40   [0.000, 5.000e-11, 1.000e-10, …]               [0.214, 0.214, 0.214, …]   [0.000, 0.000, 0.000, …]  
  run_3       ss            27   [0.000, 5.000e-11, 1.000e-10, …]               [0.103, 0.103, 0.103, …]   [0.000, 0.000, 0.000, …]  
  run_4       tt            27   [0.000, 5.000e-11, 1.000e-10, …]   [8.876e-02, 8.876e-02, 8.876e-02, …]   [0.000, 0.000, 0.000, …]  
  run_5       ff            27   [0.000, 5.000e-11, 1.000e-10, …]   [7.528e-02, 7.528e-02, 7.528e-02, …]   [0.000, 0.000, 0.000, …]  
  run_6       ss           130   [0.000, 5.000e-11, 1.000e-10, …]   [4.205e-02, 4.205e-02, 4.205e-02, …]   [0.000, 0.000, 0.000, …]  
  run_7       tt           130   [0.000, 5.000e-11, 1.000e-10, …]   [4.015e-02, 4.015e-02, 4.015e-02, …]   [0.000, 0.000, 0.000, …]  
  run_8       ff           130   [0.000, 5.000e-11, 1.000e-10, …]   [3.900e-02, 3.900e-02, 3.900e-02, …]   [0.000, 0.000, 0.000, …]

[areda0]

I agree, we should use the .ods format.

I don't think we need to report transient data here. It's meaningless to report the time reference and instantaneous values to this sheet. Only single-valued measures can go through this table. We usually extract two things from a transient sim:

• transient waveforms >> should be reported as waveforms, not numbers
• measures done on the transient data, e.g., rise time >> should be reported in the detailed sheet

For MC iterations, we're usually interested in the statistical measures (mean, std deviation..). The exact result of each iteration is useful to identify any outliers. A better way to report results across MC would be QQ-plots, which we can discuss in our next meeting, and if we want to report the numbers, then we could generate a separate sheet for each condition that lists the results for every iteration. Since we usually run MC for a single corner, typically the worst, the only variable condition would be the 'iterations' themselves.

[RTimothyEdwards]

A Q-Q plot will tell you whether the result is a normal distribution or skewed in some way, but it doesn't tell you a whole lot about the data. But I agree that transient and Monte Carlo data have to be processed in some way---Tabular output is very large and tells you nothing meaningful. Given enough data points, the equivalent list for a monte carlo result would be a listing of mean and standard deviation over sub-groups of the data vs. condition. That would show the numerical results of the distribution of INL vs. input value for the DAC example (for which INL increases when the DAC output gets close to the power rail).

Transient data should, in principle, be easier, because there is still a single value (such as frequency or pulse width or rise time) that is being measured over many sets of conditions and can be tabulated. Only monte carlo runs are more difficult, since the characterized values are not being checked for minimum/typical/maximum but are being measured as mean/standard deviation of a distribution of values.

[mole99]

@areda0 CACE now creates a CSV file alongside the Markdown summary. Let me know if this works for you as expected.

[areda0]

@mole99 Thanks for the updates. Both the .md and .csv files are working as expected, except that they only list the varying conditions, not the default conditions. While this is okay, listing all conditions would provide more insight into the data. Additionally, for reporting the pass/fail status of each value, it would be preferable to use a color code (e.g., red for failed) instead of adding a separate column for each result measure.

@RTimothyEdwards Beside the statistical measures of MC results, QQ-plots is a visually nice way to check how close the distribution is to a normal one.

[mole99]

Merging after discussion.