EmuMore's blogJekyll2018-09-18T19:49:00+02:00https://kakila.bitbucket.io/JuanPi Carbajalhttps://kakila.bitbucket.io/ajuanpi@gmail.com<![CDATA[Carbajal's Youtube channel]]>https://kakila.bitbucket.io/posts/youtube-channel2018-10-18T19:46:00+02:002018-10-18T19:46:00+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>A Youtube channel with unofficial video recordings of Juan Pablo Carbajal’s talks.</p>
<p>Check it out and subscribe!</p>
<p><a href="https://kakila.bitbucket.io/posts/youtube-channel.html">Carbajal's Youtube channel</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on October 18, 2018.</p>
<![CDATA[Patrick Troxler's master thesis]]>https://kakila.bitbucket.io/posts/patrick-troxler-msc-thesis2018-10-17T10:19:36+02:002018-10-17T10:19:36+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>Congratulations to Patrick Troxler for passing his master thesis defense!</p>
<p>Patrick used Kalman Filters to provide online inflow prediction to Water Treatment Plant in Oberengadin (under construction).
He managed to go from 0 to something that is actually working, we are sure the results will improve in a few iterations. Well done!</p>
<p>Follow the link in the title to explore the repositories with the code of his work.</p>
<p><a href="https://kakila.bitbucket.io/posts/patrick-troxler-msc-thesis.html">Patrick Troxler's master thesis</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on October 17, 2018.</p>
<![CDATA[Special Issue "Machine Learning Applied to Hydraulic and Hydrological Modelling"]]>https://kakila.bitbucket.io/posts/speacial-issue-water2018-08-01T10:19:36+02:002018-08-01T10:19:36+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>Submit to our special issue!</p>
<p>The computational power available nowadays allow us to tackle simulation challenges in hydraulic and hydrological modelling at different scales that were impossible a few decades ago.
However, even in the current situation, the time needed for these simulations is inadequate for many scientific and engineering applications, such as decision support systems, flood warning systems, design or optimization of hydraulic structures, calibration of model parameters, uncertainty quantification, real-time model-based control, etc.</p>
<p>To address these issues, the development of fast surrogate models to increase the simulation speed seems to be promising strategy: it does not require a huge investment in new hardware and software, and the same tools can be used to solve very different problems.
The field of Machine Learning offers a huge library of methods to build surrogate models, many of which have been successfully used in hydraulic and hydrological modelling.</p>
<p>In this Special Issue we would like to invite research works which incorporate Machine Learning techniques in hydraulic and hydrological modelling, such as (but not restricted to):</p>
<ul>
<li>
<p>Artificial Science, in which a relation between input and output is learned using only data, also known as data-driven methods.</p>
</li>
<li>
<p>Scientific Numerical Modelling, such as simplified numerical models, model calibration (system identification) or optimization, renormalized models, up (down)scaled models, coarse models, etc.</p>
</li>
<li>
<p>Emulation, where a fast emulator is developed based on training data derived by a slow simulator</p>
</li>
</ul>
<p>Dr. Vasilis Bellos</p>
<p>Dr. Juan Pablo Carbajal</p>
<p>Guest Editors</p>
<p><a href="https://kakila.bitbucket.io/posts/speacial-issue-water.html">Special Issue "Machine Learning Applied to Hydraulic and Hydrological Modelling"</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on August 01, 2018.</p>
<![CDATA[Machine Learning in Hydroresearch]]>https://kakila.bitbucket.io/posts/overview-ml-hydroresearch2018-07-15T10:19:36+02:002018-07-15T10:19:36+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>Is emulation just an application of machine learning to an specific problem?</p>
<figure class="center">
<img src="https://bitbucket.org/KaKiLa/emumore/wiki/img/ml_emu_model.png" alt="relation between machine learning, emulation, and mechanistic modeling" align="middle" />
<figcaption>Credit: Juan Pablo Carbajal under <a src="http://creativecommons.org/licenses/by/4.0">CCBY 4.0</a> via Emumore's Wiki.</figcaption>
</figure>
<p>The most frequent answer is probably: yes!
However, there is more to emulation than just model-free statistical learning methods.</p>
<!-- more -->
<p>Emulation make its biggest contributions, specially in terms of reduced samples (it is so expensive to run CFD simulations!) when mechanistic models can be combined with the learning algorithms.
What most people are unaware of is that most machine learning methods do not play well with models.
We put part of this discussion on our IAHR extended abstract<sup id="fnref:Carbajal2018"><a href="#fn:Carbajal2018" class="footnote">1</a></sup>.
We will publish a full version in the <a href="http://www.mdpi.com/journal/water/special_issues/Machine_Learning_Hydraulic_Hydrological">Water special issue</a> including the role of models in emulation.</p>
<p>One of EmuMore’s biggest concerns is to find methods (or develop them) that can exploit mechanistic knowledge.
We are currently looking at <a href="https://mitpress.mit.edu/books/elements-causal-inference">causal models</a>.</p>
<p><a href="#" class="btn btn-info">Stay tuned!</a></p>
<h1 id="references">References</h1>
<div class="footnotes">
<ol>
<li id="fn:Carbajal2018">
<p>Carbajal, J.P. & Bellos, V. (2018), An overview of the role of Machine Learning in hydraulic and hydrological modeling. Proc. 5th IAHR European Congress. doi:10.31224/osf.io/wgm72 <a href="#fnref:Carbajal2018" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="https://kakila.bitbucket.io/posts/overview-ml-hydroresearch.html">Machine Learning in Hydroresearch</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on July 15, 2018.</p>
<![CDATA[EmuMore at the 5th IAHR Europe Congress]]>https://kakila.bitbucket.io/posts/iahr2018-06-14T10:19:36+02:002018-06-14T10:19:36+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>EmuMore was present in the 5th IAHR Europe congress: New Challenges in Hydraulic Research and Engineering.
We organized a special session on Machine Learning.</p>
<p>Follow the link to browse the proceedings of the session.</p>
<p><a href="https://kakila.bitbucket.io/posts/iahr.html">EmuMore at the 5th IAHR Europe Congress</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on June 14, 2018.</p>
<![CDATA[Sebastiano Rusca's master thesis]]>https://kakila.bitbucket.io/posts/sebastiano-rusca-msc-thesis2018-03-23T09:19:36+01:002018-03-23T09:19:36+01:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>Congratulations to Sebastiano Rusca for passing his master thesis defense!</p>
<p>Sebastiano worked on the exploration of Gasussian Processes based emulation for flood prediction.
Among other interesting stuff, he explored the use of a classifier to decide what emulator to use (in his case the decision was only whether to use the emulator or not).</p>
<p>Follow the link in the title and check the Downloads page to get a copy of his thesis.</p>
<p><a href="https://kakila.bitbucket.io/posts/sebastiano-rusca-msc-thesis.html">Sebastiano Rusca's master thesis</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on March 23, 2018.</p>
<![CDATA[EmuMore at AGU 2017]]>https://kakila.bitbucket.io/posts/emumore-at-agu-20172017-12-13T23:38:29+01:002017-12-13T23:38:29+01:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>Dr. Vasilis Bellos presented our results on emulation of
2D shallow water equations used for the simulation of the Tous dam break.
The publication is coming soon, with data and scripts!</p>
<p><a href="https://kakila.bitbucket.io/posts/emumore-at-agu-2017.html">EmuMore at AGU 2017</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on December 13, 2017.</p>
<![CDATA[Computational effort and emulation error]]>https://kakila.bitbucket.io/posts/computational-effort-and-emulation-error2017-10-04T10:00:00+02:002017-10-04T10:00:00+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>There is nothing better to force a new perspective on the work we do than the
questions from other people.
In this post we revisit the idea of emulation inspired by some surgical questions
formulated by <a href="http://www.eawag.ch/en/aboutus/portrait/organisation/staff/profile/kris-villez/show/">Kris Villez</a>,
aiming at understanding the relation between an emulator and the simulator it is
based on.</p>
<p>These are the questions we will discuss:</p>
<ol>
<li>
<p>Can my emulator make exact predictions of my simulator output for any input?</p>
</li>
<li>
<p>If my emulator makes exact predictions, will it be faster than my simulator?</p>
</li>
<li>
<p>How small should my emulation error be?</p>
</li>
</ol>
<figure class="center">
<img src="https://bitbucket.org/KaKiLa/emumore/raw/tip/doc/img/emulation_errorvscost.png" alt="Emulation error vs computational effort" align="middle" />
<figcaption>Credit: CCBY4.0 based on Kris Villez' original diagram</figcaption>
</figure>
<!-- more -->
<h2 id="figure-summary">Figure summary</h2>
<p>The figure above summarizes the discussions in this post.
They involve a simulator (“current simulator” in the figure), and a function(al) used to compute some low dimensional output which is the relevant signal for the task at hand, i.e. we are looking for an input-output relation (where output is not the output of the simulator, but of some dimension reducing function(al)).
An emulator is built to map inputs to this low dimensional output.</p>
<p>Lets describe the different items in the figure above.</p>
<h3 id="y-axis-emulation-error">Y-axis: Emulation error</h3>
<p>The vertical axis of the figure indicates the emulation error, that is the error we obtain when we test our emulator on new inputs and compare the emulation result with the output of the simulator.
In machine learning this is usually called the <em>test</em> or <em>generalization</em> error.</p>
<h3 id="x-axis-computational-effort">X-axis: Computational effort</h3>
<p>The horizontal axis shows how much computation we need to get a result.
The current simulator defines the computational budget, i.e. an emulator that needs a similar computational effort is not interesting.
Computational effort is usually measured as the amount of time we have to wait to get an output from the simulator (runtime), but it could be measured in other terms relevant for the application at hand, e.g. electrical power consumption.</p>
<p>In this axis we also marked the effort required by an ideal implementation of the simulator (“ideal simulator”): the ultimately simplified simulator.
For very complex systems this ideal simulator tends to be the system itself.</p>
<h3 id="simulator-simplification">Simulator simplification</h3>
<p>There are two lines emanating from the wrongest emulator, i) “Simulator simplification” and, ii) “Exact prior”.
We will explain the latter in subsequent paragraphs.
The former, “Simulation simplification”, refers to the situation in which, by using the data available, we are able to build an emulator that is faster than the current simulator and is also exact.
This is seldom possible using the data-driven methods commonly used in emulation, and it also depends heavily on the nature of the simulated process.
Such situation is more related to the optimization of algorithms, data structures, and hardware.
We will not directly discuss this situation here, but see the part on “Exact emulation”.</p>
<h3 id="task-error-tolerance">Task error tolerance</h3>
<p>Another important aspect of the figure is the horizontal green line labeled “Task error tolerance”.
That is, the task might be accomplish with the same performance even if the emulator doesn’t match the simulator+function(al) results exactly.
For discussions on this you can read about approximate computing <sup id="fnref:Han13"><a href="#fn:Han13" class="footnote">1</a></sup>, but the intuition is quite straight forward: take <script type="math/tex">\vec{x}</script> as the output of your simulator+function(al) and say that the relevant signal is a linear combination of those outputs, i.e <script type="math/tex">\vec{n} \cdot \vec{x} = y</script>.
Now consider an emulator that produces a wrong output <script type="math/tex">\vec{z} = \vec{x} + \vec{\epsilon}</script>.
Any error that fulfills the equality <script type="math/tex">\vec{n} \cdot \vec{\epsilon} = 0</script> will produce exact results on <script type="math/tex">\vec{y}</script>.</p>
<p>In other words being wrong doesn’t imply that the error will propagate to the relevant signals.
Hence the task, expressed above as a linear combination of the outputs, might give us some room for error and the possibility of saving some computational resources.
This is one of the reasons why emulation is very useful when the task reduces the dimension of the simulator outputs.
The reduction might create an effective “null-space” in which we can place the emulation error.
I write “null-space” in quotes, because I am abusing a concept from linear algebra, but the ideal holds.
In thermodinamics we would speak of entropy to quantify the size of the allowed error set (a sort of partition function).</p>
<h3 id="emulators">Emulators</h3>
<p>Here the people conversant with parametric and non-parametric methods will have to excuse my hand-waving explanations.
With the aim of making this article valuable for the non-experts some correctness was thrown away.</p>
<p>In most cases, improving our emulator by reducing the error increases the computational effort.
For example if we are using a fixed basis of functions to regress the simulation results (a parametric method, e.g. <a href="https://en.wikipedia.org/wiki/Generalized_Fourier_series">generalized Fourier series</a>, <a href="https://en.wikipedia.org/wiki/Polynomial_chaos">polynomial chaos expansion</a>, or <a href="https://en.wikipedia.org/wiki/Wiener_series">Wiener/Volterra series</a>, etc), the emulator can get better by adding new basis functions/kernels and adapting already existing ones (e.g. <a href="https://en.wikipedia.org/wiki/Basis_pursuit">basis pursuit</a> (<a href="https://en.wikipedia.org/wiki/Basis_pursuit_denoising">denoising</a>), <a href="https://en.wikipedia.org/wiki/Matching_pursuit">matching pursuit</a>, <a href="https://en.wikipedia.org/wiki/Machine_learning#Sparse_dictionary_learning">dictionary learning</a>, etc.).</p>
<p>In the case of non-parametric methods such as <a href="http://www.gaussianprocess.org/">Gaussian Processes</a>, this could be achieved by adding more data points.</p>
<h3 id="optimal-emulator">Optimal emulator</h3>
<p>When the emulation error crosses the level defined by the task tolerance, we have found the optimal emulator.
This sounds good, but the problem is that most of the time we cannot calculate the task error tolerance, or the magnitude of the error is not actually a good indicator.
Take our example above: two emulator with the same error magnitude <script type="math/tex">\Vert\vec{\epsilon}\Vert</script> are not necessarily both perpendicular to <script type="math/tex">\vec{n}</script>.</p>
<h3 id="exact-prior-and-prior-mismatch">Exact prior and prior mismatch</h3>
<p>When should we expect to achieve a zero error?
Only when we are able to guess/find the correct representation of the input-output relation we are trying to learn.</p>
<p>Lets get away from math for a second to illustrate the negative case.
What I wrote above is something like: if you are using cats features to represent birds, you can expect to get close but not that close, and you will probably never get a good representation of a bird.
More mathematically, we could say that if the relation we are trying to learn is not contained in the “space” generated by the chosen basis then we can hope for a small but not negligible error.
A common example of this phenomena is given by the difficulty in approximating a discontinuous function by a finite series of continuous functions, i.e. <a href="https://en.wikipedia.org/wiki/Gibbs_phenomenon">Gibbs phenomena</a>.</p>
<p>The positive case is embodied by the <a href="https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem">Nyquist–Shannon sampling theorem</a>, in which a signal is perfectly recovered only using a finite set of observations.</p>
<p>All boils down to the same thing, if your prior knowledge (your assumptions about the input-output relation) is good, then you can hope for a perfect emulation.
Furthermore, if your emulation method is cheaper than your simulator+function(al) you will get a sort of simulator simplification because your emulator will be cheaper to evaluate than your simulator and still produce perfect outputs.
However, I wouldn’t expect that this happens very often in practice…but it is not impossible.</p>
<h2 id="on-exact-emulation">On exact emulation</h2>
<p>To illustrate the possibility of exact emulation, I need you to use your imagination.
Imagine that there is an alien race that has not yet discovered <a href="https://en.wikipedia.org/wiki/Hooke%27s_law">Hooke’s law</a> of linear elasticity.
But they are quite advance in terms of molecular simulations; they can build simulators that calculate forces between many many atoms.
Among these aliens there is a great inventor who is developing a device that when deformed produces a force.</p>
<p><img src="https://www.researchgate.net/profile/Lilian_Davila2/publication/270290741/figure/fig6/AS:295201777176581@1447392996279/Figure-7-Snapshot-image-from-LAMMPS-MD-silica-helical-nanostructure-nanoribbon-tensile.png" alt="Molecular spring model" width="50%" height="50%" align="left" style="margin: 0px 10px 0px 0px" /></p>
<p>The inventor sets up a simulator using their molecular technique.
The model describes the shape of the device for any deformation and it calculates the forces between all the molecules in the device.
This way the inventor can simulate the force generated at each deformation.
Their simulator is, of course, computing a lot more than just that force value.
It is indeed tracking the positions and velocities of all the molecules in the device and calculating all the inter molecular forces.
This is a very costly simulator indeed!</p>
<p>To accelerate the molecular simulator, the inventor produces a dataset of small deformations and the correspoding forces.
Using this dataset they build an interpolating function, i.e. an emulator, that will provide the force for any deformation.
It turns out that the interpolant is just a linear function, the emulation is exact and it is also many times faster than the simulator! The inventor just discovered Hooke’s law.</p>
<h3 id="how-is-this-possible">How is this possible?</h3>
<p>One the one hand, the emulator is not calculating all the things the simulator is calculating.
On the other hand, the total force generated by the device is an aggregation of all the intermolecular forces and although each force has a very complicated relation with the deformation the total force does not.
So two ingredients are at play, i) the emulator does not compute everything the simulator does, and ii) the observed output (force) maintains a simple relationship with the input (deformation).
That is, we are throwing information away and, with some luck, the relation between the remaining magnitudes can be expressed in simpler terms.</p>
<p>When these two ingredients are present you can hope to find a very good emulator.
In physics people use the word <a href="https://www.complexityexplorer.org/tutorials/67-introduction-to-renormalization">renormalization</a> to refer to the process of throwing irrelevant information and the calculation of the new effective model (also called effective theory or coarse-grained model).
The method is fully formalized for quantum fields, but the ideas are spreading to general models <sup id="fnref:Wolpert14"><a href="#fn:Wolpert14" class="footnote">2</a></sup> <sup id="fnref:Crutchfield14"><a href="#fn:Crutchfield14" class="footnote">3</a></sup>.
I expect emulation will profit a lot from these developments.</p>
<h1 id="references">References</h1>
<div class="footnotes">
<ol>
<li id="fn:Han13">
<p>Han, J., & Orshansky, M. (2013). Approximate computing: An emerging paradigm for energy-efficient https://doi.org/10.1109/ETS.2013.6569370 <a href="#fnref:Han13" class="reversefootnote">↩</a></p>
</li>
<li id="fn:Wolpert14">
<p>Wolpert, D. H., Grochow, J. A., Libby, E., & DeDeo, S. (2014). A framework for optimal high-level descriptions in science and engineering—preliminary report. arXiv preprint arXiv:1409.7403. <a href="#fnref:Wolpert14" class="reversefootnote">↩</a></p>
</li>
<li id="fn:Crutchfield14">
<p>Crutchfield, J. P., James, R. G., Marzen, S., & Varn, D. P. (2014). Understanding and Designing Complex Systems: Response to” A framework for optimal high-level descriptions in science and engineering—preliminary report”. arXiv preprint arXiv:1412.8520. <a href="#fnref:Crutchfield14" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="https://kakila.bitbucket.io/posts/computational-effort-and-emulation-error.html">Computational effort and emulation error</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on October 04, 2017.</p>
<![CDATA[Wiki renders math]]>https://kakila.bitbucket.io/posts/wiki-renders-math2017-09-20T08:10:34+02:002017-09-20T08:10:34+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>If you have peeked into our <a href="https://bitbucket.org/KaKiLa/emumore/wiki/Home">wiki</a> you might have noticed that MathJax was not rendering, sorry for that!
This was due to a bug in the BitBucket platform that now is fixed: kudos for the Atlassian team!</p>
<p>Now you should get rendered math even if you do not have a BitBucket account, <a href="https://bitbucket.org/KaKiLa/emumore/wiki/adimensional_shallow_water.rst">give it a try</a>.</p>
<p><a href="https://kakila.bitbucket.io/posts/wiki-renders-math.html">Wiki renders math</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on September 20, 2017.</p>
<![CDATA[Biokinetic Models]]>https://kakila.bitbucket.io/posts/biokinetic-models2017-08-14T14:03:15+02:002017-08-14T14:03:15+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>New process technologies that enable the shift from conventional biological
wastewater treatment processes to resource recovery systems
are matched by an increasing demand for predictive capabilities.
Mathematical models are excellent tools to meet this demand, but
they runtime still hinders parameter indetification and real-time control.</p>
<!-- more -->
<p>In collaboration with <a href="http://www.eawag.ch/en/aboutus/portrait/organisation/staff/profile/kris-villez/show/">Kris villez</a>
from the Department Process Engineering at Eawag, we are developing
emulators to accelerate the different modules that compose the simulators
of Nitrification bioreactors.</p>
<p>One important step in the simulation of these systems is the calulation of
the pH based on the state of the reactor<sup id="fnref:Masic2017"><a href="#fn:Masic2017" class="footnote">1</a></sup><sup id="fnref:FloresAlsina2015"><a href="#fn:FloresAlsina2015" class="footnote">2</a></sup>.
The pH is estimated by a root finding process that could be accelerated
inducing an overall speed up of the simulator.</p>
<p>We have been working on simplified versions of the root finding module.
The first test was in <a href="https://bitbucket.org/KaKiLa/emumore/wiki/Bioreactor_CS.rst#!case-study-bioreactor">2-dimensions</a>
and a later one in <a href="https://bitbucket.org/KaKiLa/emumore/wiki/Bioreactor_CS_2.rst#!case-study-bioreactor-4d-input">4-dimensions</a>.
The results are promising and next we will tackle the 12-dimensional case.</p>
<figure class="center">
<img src="https://bitbucket.org/KaKiLa/emumore/wiki/img/bioreactor4d_GP.png" alt="pH as function of 4 inputs" align="middle" />
<figcaption>Credit: Juan Pablo Carbajal under <a src="http://creativecommons.org/licenses/by/4.0">CCBY 4.0</a> via Emumore's Wiki.</figcaption>
</figure>
<p>We will make the final dataset publicly available.</p>
<p><a href="#" class="btn btn-info">Stay tuned!</a></p>
<h1 id="references">References</h1>
<div class="footnotes">
<ol>
<li id="fn:Masic2017">
<p>Masic, A., Srinivasan, S., Billeter, J., Bonvin, D., & Villez, K. (2017). Identification of Biokinetic Models using the Concept of Extents. Environmental Science & Technology. http://doi.org/10.1021/acs.est.7b00250 <a href="#fnref:Masic2017" class="reversefootnote">↩</a></p>
</li>
<li id="fn:FloresAlsina2015">
<p>Flores-Alsina, X., Kazadi Mbamba, C., Solon, K., Vrecko, D., Tait, S., Batstone, D. J., Jeppsson, U., Gernaey, K. V. (2015). A plant-wide aqueous phase chemistry module describing pH variations and ion speciation/pairing in wastewater treatment process models. Water Research, 85, 255–265. http://doi.org/10.1016/j.watres.2015.07.014 <a href="#fnref:FloresAlsina2015" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="https://kakila.bitbucket.io/posts/biokinetic-models.html">Biokinetic Models</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on August 14, 2017.</p>
<![CDATA[Emulation Talk at ZHAW Winterthur, Switzerland]]>https://kakila.bitbucket.io/posts/emulation-talk-at-zhaw-winterthur-switzerland2017-08-14T13:37:22+02:002017-08-14T13:37:22+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>We will presenting some of our work at the
2nd European COST Conference on Mathematics for Industry in Switzerland,
Artificial Intelligence in Industry and Finance.</p>
<p><a href="https://kakila.bitbucket.io/posts/emulation-talk-at-zhaw-winterthur-switzerland.html">Emulation Talk at ZHAW Winterthur, Switzerland</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on August 14, 2017.</p>
<![CDATA[Rotated Principal Components. Interpretability in the Eye of the Beholder]]>https://kakila.bitbucket.io/posts/rotated-principal-components-interpretability-in-the-eye-of-the-beholder2017-07-20T09:42:20+02:002017-07-20T09:42:20+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>In the sciences, it is increasingly more common to deal with high-dimensional datasets.
In some fields like meteorology this was always the case.
Many tools have been released to perform “dimensionality reduction”, which enables us to interpret and handle these huge datasets more easily.
One of the most popular methods is Principal Components Analysis (PCA) (we have talked about its <a href="/emumore/posts/polyonymous-algorithms.html">many guises</a>), but it has been criticized.
Here we make a short and partial review of “Rotation of principal components” by M. B. Richman, a highly cited article from the 80’s popular in earth sciences …</p>
<figure class="center">
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/f/f5/GaussianScatterPCA.svg/240px-GaussianScatterPCA.svg.png" alt="PCA of a multivariate Gaussian distribution" align="middle" />
<figcaption>Credit: CCBY4.0 Nicoguaro at Wikimedia commons</figcaption>
</figure>
<!-- more -->
<p>In his article, Richman <sup id="fnref:Richman1986"><a href="#fn:Richman1986" class="footnote">1</a></sup> reviewed the uses an applications of matrix factorizations (he called them principal components decompositions).
Therein he criticized properties of the method and suggested that “meaningful” decompositions are obtained via linear (affine) transformations of the components.
His article was cited more than 1600 times, and I think is was quite influential in earth sciences.
Others have also published their opinion, for example <sup id="fnref:Jolliffe1987"><a href="#fn:Jolliffe1987" class="footnote">2</a></sup> discusses several issues in Richman criticism.
From nowadays stand point of view the Richman’s criticism should be seen more as a desiderata on the properties of the extracted components rather as a critic on the method.
As such, one could discuss if the desiderata is actually relevant or not, but the method remains unscathed.</p>
<p>Richman mentioned several properties of principal components decomposition as undesired, e.g. that the decomposition is only unique after a normalization is provided, that the method depends on local properties of the data (subdomain stability), etc… check the article for the whole thing.</p>
<p>In this post I go thought the aspects that I find most relevant for the EmuMore project.</p>
<h2 id="the-basics">The basics</h2>
<p>Lets say we measured a time dependent magnitude on a mesh in 3D space during a time-span of duration <script type="math/tex">T</script>, we then have a dataset associating each point in 3D space with a real signal in time, i.e. <script type="math/tex">\mathbb{R}^3 \rightarrow \mathbb{R}([0,T])</script>, or equivalently <script type="math/tex">\mathbb{R}^3\times[0,T] \rightarrow \mathbb{R}</script>.
We could represent this data also as a function (if we know there is a functional relationship), <script type="math/tex">y(\vec{r},t)</script>.</p>
<p>Here comes the idea: can we represent the function as the combination of other functions? (a core question in <a href="https://en.wikipedia.org/wiki/Kolmogorov–Arnold_representation_theorem">Hilbert’s 13th problem</a> (1900), in particular we look for a factorization of the form</p>
<script type="math/tex; mode=display">y(\vec{r},t) = \sum_{i=1}^{M} {u_i(\vec{r}) w_i(t)}</script>
<p>which allows us to think of the data as composed by some spatial modes <script type="math/tex">u_i(\vec{r})</script> that are combined over time with weights <script type="math/tex">\vec{w}(t)</script>.
If we manage to find this factorization with a small number of modes then we will be able to understand our data better, by just understanding the way the modes are combined (check <a href="https://en.wikipedia.org/wiki/Normal_mode">normal models</a> in physics or <a href="https://en.wikipedia.org/wiki/Eigenfunction">eigenfunctions</a> in mathematics)</p>
<p>If we have discretized space and time (mesh of measurements every so many seconds), we can then refer to the set <script type="math/tex">\mathbb{R}^3\times[0,T]</script> by the indexes of elements in the mesh instead of their value.
The data can then be stored as a matrix <script type="math/tex">Y \in \mathbb{R}^{n_x n_y \times n_t}</script>, where <script type="math/tex">n_x,n_y,n_t</script> are the number of points in each direction of the mesh.
With this view, the data becomes a matrix <script type="math/tex">Y</script> and the factorization above would look like a matrix-vector product</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
Y_i(t_k) &= \sum_{m=1}^{M} U_{ij} w_j(t_k)\\
Y(t_k) &= U \vec{w}(t_k) \quad U \in \mathbb{R}^{n_x n_y \times M}\\
\end{align*}$ %]]></script>
<p>In the article about <a href="/emumore/posts/polyonymous-algorithms.html">factorization algorithms</a> we showed that SVD provides</p>
<script type="math/tex; mode=display">Y = U D V^\top</script>
<p>with <script type="math/tex">U^\top U = I</script> and <script type="math/tex">V^\top V = I</script>, which is what the factorization needs if we define <script type="math/tex">w = D V^\top</script>.
The columns of the matrix <script type="math/tex">U</script> form a basis in which the data is decomposed and <script type="math/tex">w</script> are the components of the data in that basis.</p>
<p>If we apply an invertible linear transformation to the basis, the data is projected with new components <script type="math/tex">w^\prime</script>,</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
Y &= (T U) w^\prime\\
U^\top T^{-1} U w &= w^\prime
\end{align*} %]]></script>
<p>Meaning that we can always transform the basis (without loosing information) and obtain new components.</p>
<p>The set of all invertible transformation is infinite and this made Richman <sup id="fnref:Richman1986:1"><a href="#fn:Richman1986" class="footnote">1</a></sup> talk about an infinite level of degeneracy in the decomposition.
This is a fact from linear algebra: the set of bases of linear vector spaces is infinite, all mathematically equivalent.
However, if we add desired properties to the components or the basis, we can pick one out of the infinite set. For example, PCA maximizes the standard deviation of the components.</p>
<p>In the last equation, if we require that the decomposition produces a basis <script type="math/tex">U</script> which is orthonormal with respect to <script type="math/tex">T^{-1}</script> (instead of the usual scalar product <script type="math/tex">U^\top U = I</script>) then we get <script type="math/tex">w = w^\prime</script>.
If we can decompose the data with this constraint (the so called normalization constraint in Richman’s) we obtain the decomposition in the transformed basis.</p>
<p>Enter POD …</p>
<h2 id="transformation-as-scalar-products">Transformation as scalar products</h2>
<p>When the matrix <script type="math/tex">T^{-1}</script> has only (non-negative)positive eigenvalues (i.e. positive (semi-)definite) it can be interpreted as a <a href="https://en.wikipedia.org/wiki/Gramian_matrix">Gramian matrix</a> or a scalar product.
This is how PCA is implemented in many fields that deal with differential equations and their numerical solutions <sup id="fnref:Volkwein13"><a href="#fn:Volkwein13" class="footnote">3</a></sup>, the given name of the method in these fields is usually Proper Orthogonal Decomposition (POD).
In this view, the condition of orthonormality <script type="math/tex">U^\top R^{-1} U = I</script> can be implemented directly into the SVD decomposition,</p>
<script type="math/tex; mode=display">U^\top W U = I = \left(\sqrt{W} U\right)^\top \left(\sqrt{W} U\right) = U_W^\top U_W</script>
<p>which means that applying SVD to the weighted data <script type="math/tex">\sqrt{W} Y</script> and then solving <script type="math/tex">\sqrt{W} U = U_W</script> will produce the desired basis.</p>
<p>This generalized PCA is implemented in the function <a href="https://bitbucket.org/KaKiLa/emumore/src/tip/src/mfiles/pod.m">pod.m</a> in the repository.</p>
<h2 id="to-transform-or-not-to-transform">To transform or not to transform</h2>
<p>Bases of linear vector spaces are equivalent; citing Richman “<em>…[referring to two bases] have the same properties and are indistinguishable</em>”.
So there is no intrinsic “better” , despite of what the discourse in his article seems to indicate.</p>
<p>The goodness of each representation lies in the researcher’s use of the decomposition.
In his article he argues for decompositions that are aligned with how explanations of phenomena are constructed in his field, i.e. he likes components and bases that allows him to generate explanations with the usual jargon and concepts.
Hence he struggles (and reviews somebody else’s struggle) to represent this in terms of linear transformations and normalizations of the components and bases.</p>
<p>If the objective is just dimensionality reduction than any basis would do, you can stick to the output of SVD.
If the objective also includes interpretability of the results with widespread concepts or the if basis should embody some prior knowledge we have about the phenomena generating the data, then transforming the basis and components provided by SVD is highly desired (probably necessary).
Transformation are also useful if further algorithms applied to the components or basis have properties that suit certain type of data.
In that case it is desired to choose a decomposition that matches the properties of those algorithms, e.g. if we want to parallelize further analyses involving the components we might want to extract sparse representations or statistically independent components.</p>
<h2 id="subdomain-stability">Subdomain stability</h2>
<p>One of the concerns raised in Richman’s article is that extraction of components
is not stable if one performs the analysis in a subdomain of the original domain.
Attributing subdomain stability to the extracted components reflects prior information
that the analyst has about the process generating the data.
It is, by no means, a deficiency of the decomposition.
Indeed, subdomain stability is expected only on specific types of data.
Lets briefly describe types of data with and without subdomain stability:</p>
<ul>
<li>
<p><strong>Scale free data</strong>
When the subdomain is built by zooming in the original domain, stability
will occur if the properties of the data are the same (very similar) at different
scales.
Here is an excellent video demonstrating this properties in critical Ising models</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/MxRddFrEnPc?rel=0&showinfo=0" frameborder="0" allowfullscreen></iframe>
<p>For scale-free data, the extracted components in the subdomain will be compatible with the
ones in the original domain.
Many types of noise (and chaotic signals) are also self-similar or scale invariant.</p>
</li>
<li>
<p><strong>Bifurcation data</strong>
Many dynamical systems show extreme changes in their behavior when parameters
change.
In an spatially distributed scenario, even if the same dynamical process is
generating the data, parameters (like, topography, temperature, humidity, etc…) depend on the position in the domain.
If the variation of these parameters is not uniform then components of data from
subdomain will reflect these local variability , e.g. the density of bees
is concentrated more in agricultural or green areas but not everywhere, like in
deserts or cities, hence a process regulated by bee population will produce different
data in these subdomains, specially if it undergoes a bifurcation.</p>
<p>Here is an example of the behavior of a car-trailer system when mass distribution is changed:</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/i2fkOVHAC8Q?rel=0&showinfo=0&muted=1" frameborder="0" allowfullscreen></iframe>
<p>Similarly, systems that undergo drastic change in behavior based on their states,
known as Hybrid/Switching systems, will show different components in different
regions of the observed data.</p>
</li>
<li>
<p><strong>Data with local variation</strong>
If the characteristics of the signal are defined by their position in the
domain (as in bifurcations) the components are not expected to be subdomain invariant.
I have written the script <a href="https://bitbucket.org/KaKiLa/emumore/src/tip/src/mfiles/s_pc_subdomain.m">s_pc_subdomain.m</a> (here its <a href="https://kakila.bitbucket.io/pc_subdomain/s_pc_subdomain.html">html report</a>) to
illustrate this based on signals that look like this:</p>
</li>
</ul>
<figure class="center">
<img src="https://kakila.bitbucket.io/pc_subdomain/s_pc_subdomain-1.png" alt="Signals with local characteristics" align="middle" />
<figcaption>Signals used in the demonstration script (see text). The black line indicates the two regions where the signals have different characteristic length scales</figcaption>
</figure>
<p>The results show that the subdomain components are very often
considerably different from the ones in the whole domain (80 out of 100 have relative L2 difference higher than 0.5).</p>
<h1 id="references">References</h1>
<div class="footnotes">
<ol>
<li id="fn:Richman1986">
<p>Richman, M. B. (1986). Rotation of principal components. Journal of Climatology, 6(3), 293–335. http://doi.org/10.1002/joc.3370060305 <a href="#fnref:Richman1986" class="reversefootnote">↩</a> <a href="#fnref:Richman1986:1" class="reversefootnote">↩<sup>2</sup></a></p>
</li>
<li id="fn:Jolliffe1987">
<p>Jolliffe, I. T. (1987). Rotation of principal components: Some comments. Journal of Climatology, 7(5), 507–510. http://doi.org/10.1002/joc.3370070506 <a href="#fnref:Jolliffe1987" class="reversefootnote">↩</a></p>
</li>
<li id="fn:Volkwein13">
<p>Volkwein, S. (2013). Proper orthogonal decomposition: Theory and reduced-order modelling. Lecture Notes, University of Konstanz. http://www.math.uni-konstanz.de/numerik/personen/volkwein/teaching/POD-Book.pdf <a href="#fnref:Volkwein13" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="https://kakila.bitbucket.io/posts/rotated-principal-components-interpretability-in-the-eye-of-the-beholder.html">Rotated Principal Components. Interpretability in the Eye of the Beholder</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on July 20, 2017.</p>
<![CDATA[Model Order Reduction and Emulation]]>https://kakila.bitbucket.io/posts/model-order-reduction-and-emulation2017-07-13T11:18:23+02:002017-07-13T11:18:23+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>So you have your nifty numerical simulator that captures all the details of the complex device
you want to model, everything! You got your <a href="https://en.wikipedia.org/wiki/Digital_twin">digital twin</a>, and for a while everything is OK.</p>
<h2 id="beyond-reality">Beyond reality</h2>
<p>There comes the day in which you want to use your simulator in ways that go beyond the constraints imposed by the physical world … well, that is the reason why we build simulators, isn’t it?</p>
<!-- more -->
<p>You want to:</p>
<ol>
<li>optimize the design of your device</li>
<li>find out the values of parameters of your device that are hard or impossible to measure (i.e. system identification)</li>
<li>use the model to predict the behavior of your device under different situations and control it in realtime.</li>
</ol>
<p>At this point you realize that your nifty model is just too slow. Maybe a single simulation is many times faster than reality, lets say a simulation of your device running for months takes a few minutes, but applications 1.-3. require thousands of simulations, and even a runtime of a few minutes make those applications impractical.</p>
<p>So, what do you do? First, do not panic, you are not alone.
This problem is quite common and there is a whole <a href="https://morwiki.mpi-magdeburg.mpg.de/morwiki">field of applied science that deals with it</a>.
In a nutshell: you need to speed-up your model by making it specific to the problem at hand.</p>
<h2 id="problem-specification">Problem specification</h2>
<p>To speed-up your simulator you need to check a few properties of the usage you have planned for it.
In particular you want to check if you actually need all the details that your simulator provides.
For example, a simulator based on the <a href="https://en.wikipedia.org/wiki/Finite_element_method">Finite Element Method</a> (FEM) will produce a huge amount of values that you might not need to actually get the information you need for your application.</p>
<p>Loosely speaking, numerical simulators used in applications like the ones enumerated in the previous section have a fan-out/fain-in structure.
Let an image explain this idea:</p>
<figure class="center">
<img src="https://bitbucket.org/KaKiLa/emumore/wiki/img/Simulator.png" alt="Fan-out/fan-in in numerical simulators" align="middle" />
<figcaption>Fan-out/fan-in in numerical simulators</figcaption>
</figure>
<p>The <strong>fan-out</strong> part comes from the way inputs, e.g. boundary conditions, actuations, parameter values, etc., are mapped to internal states of the simulator (usually many … many!).
Continuing with the FEM example, a few parameters defining the problem will produce hundreds or thousands of nodal values.
Your simulator spends precious time keeping track of this plethora of values.</p>
<p>The <strong>fan-in</strong> part is given by the application at hand.
Usually the application does not need the whole set of internal states, e.g. you might be interested in the signal produced by a sensor obtained from the average of states over a small region of your domain.
In such (pervasive!) situations the quantity of output values is much smaller than the dimension (order) of the internal states of the simulator.
Your simulator, by the way, is establishing a relation between the inputs and the outputs of your problem.</p>
<p>The natural question is then: can we find a smaller model that establishes the same (or a similar) relation between inputs and outputs?
And the answer is generally yes, such reduced model exists.</p>
<h2 id="reduced-models">Reduced models</h2>
<p>The general way of reducing a model, and well suited for models based on differential equations (we will mainly refer to these), is via <a href="https://en.wikipedia.org/wiki/Model_order_reduction">Model Order Reduction</a> (MOR).
MOR methodologies use information from your simulator and a set of simulations (snapshots)
to create a reduced model that behaves very much like your detailed simulator.</p>
<p>MOR methodologies based on projections build a model with a structure derived from the structure of your simulator and with parameters adapted to reproduce the outputs of your simulator:</p>
<figure class="center">
<img src="https://bitbucket.org/KaKiLa/emumore/wiki/img/MOR.png" alt="Model order reduction" align="middle" />
<figcaption>Model order reduction</figcaption>
</figure>
<p>These methodologies give you some information about the internal states of the detailed simulator, but “blurred out”.
It is like looking at a picture with your eyes ajar: you still see the picture, but textures and patterns with small details are blurred out (for the physicist out there: it is a kind of <a href="https://en.wikipedia.org/wiki/Coarse-grained_modeling">coarse-graining</a>).
The reduced model, having much fewer states and capturing the most important aspects of the input-output relation, spends less time in internal calculations and offers a speed-up.</p>
<p>Ey! Why not we take this to the extreme and get rid of all internal states? Enter emulation…</p>
<h2 id="emulators">Emulators</h2>
<p>Emulation takes MOR to its natural limit and uses only snapshots of your simulations to learn the input-output relation.</p>
<figure class="center">
<img src="https://bitbucket.org/KaKiLa/emumore/wiki/img/Emulation.png" alt="Emulation" align="middle" />
<figcaption>Emulation</figcaption>
</figure>
<p>Emulation is a regression problem, even more, it is an <a href="http://lavidasegunjuanpi.blogspot.ch/2016/09/inter-extra-and-intra-polation.html">interpolation</a> and falls within the realm of <a href="https://work.caltech.edu/telecourse.html">Machine Learning</a> and <a href="http://www.cambridge.org/catalogue/catalogue.asp?isbn=9780521843355">Scattered Data Approximation</a><sup id="fnref:Wendland05"><a href="#fn:Wendland05" class="footnote">1</a></sup>.
Emulation will give you the highest speed-up attainable, but it is not always feasible.</p>
<p>If emulation exploits information from your simulator it is called <em>mechanistic</em> or invasive, and if it does not then is called <em>data-driven</em> (see <a href="https://arxiv.org/abs/1609.08395">this article</a><sup id="fnref:Carbajal16"><a href="#fn:Carbajal16" class="footnote">2</a></sup> for a discussion).
Having information about the simulator that generated the snapshots helps guessing the input-output relation far away from the given data.</p>
<p>Data-driven emulation is simpler to apply than mechanistic emulation and MOR.
That is why you should try it first of all.</p>
<h1 id="recap">Recap</h1>
<p>MOR and emulation, the extreme case, can help you with your speed problem.
But there is no free lunch: to create any of these faster models, you will still need
to run some simulations and then apply methods for the creation of the effective model.</p>
<p>Data-driven emulation uses no information about your simulator (besides the data), therefore the error of its predictions for inputs far away from the data is uncontrollable.
Mechanistic emulation migth produce better trends outside the data scope, since it uses some information about the simulator structure.
This is also the case with MOR.</p>
<p>An important difference between MOR and Emulation is that the former allows you to represent the
(blurred) internal states of your simulator, while the latter does not.</p>
<h2 id="references">References</h2>
<div class="footnotes">
<ol>
<li id="fn:Wendland05">
<p>Holger Wendland (2005). Scattered Data Approximation. Cambridge Monographs on Applied and Computational Mathematics 17. <a href="#fnref:Wendland05" class="reversefootnote">↩</a></p>
</li>
<li id="fn:Carbajal16">
<p>Carbajal, J. P., Leitão, J. P., Albert, C., & Rieckermann, J. (2017). Appraisal of data-driven and mechanistic emulators of nonlinear simulators: The case of hydrodynamic urban drainage models. Environmental Modelling & Software, 92, 17–27. http://doi.org/10.1016/j.envsoft.2017.02.006 <a href="#fnref:Carbajal16" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="https://kakila.bitbucket.io/posts/model-order-reduction-and-emulation.html">Model Order Reduction and Emulation</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on July 13, 2017.</p>
<![CDATA[Groundwater Recharge Rate]]>https://kakila.bitbucket.io/posts/ground-water-recharge-rate2017-06-12T10:24:31+02:002017-06-12T10:24:31+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>After a fruitful meeting with <a href="http://www.eawag.ch/en/aboutus/portrait/organisation/staff/profile/christian-moeck/show/">Christian Möck</a> from the Department of <a href="http://www.eawag.ch/en/department/wut/">Water Resources & Drinking Water</a> at Eawag, we agreed to collaborate on building an emulator for a physics based simulator of <a href="https://en.wikipedia.org/wiki/Groundwater_recharge">Groundwater recharge rates</a>.</p>
<figure class="center">
<img src="https://upload.wikimedia.org/wikipedia/commons/8/80/Surface_water_cycle.svg" alt="Surface water cycle" align="middle" />
<figcaption>Credit: M. W. Toews under <a src="http://creativecommons.org/licenses/by/4.0">CCBY 4.0</a> via Wikimedia Commons.</figcaption>
</figure>
<!-- more -->
<p>Groundwater recharge is the primary hydrologic process through which water enters the soil and refills aquifers (so important!).
Understanding this process is fundamental to the sustainable management of groundwater since the recharge magnitude directly affects the amount of water that can be extracted from aquifers.</p>
<p>Therefore there has been many developments in the area of recharge modeling (<a href="http://www.swap.alterra.nl/">SWAP</a>, <a href="https://water.usgs.gov/software/lists/groundwater">MODFLOW</a>, etc) and a major issue is the calibration of these models.
In this collaboration we will build an emulator for an in-house simulator with the aim of running a full-fledged calibration using <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo methods</a>.</p>
<p>One of the fundamental models for the dynamics of recharge is the <a href="https://en.wikipedia.org/wiki/Richards_equation">Richards equation</a></p>
<script type="math/tex; mode=display">\frac{\partial \theta}{\partial t}= \frac{\partial}{\partial z} \left[ K(\theta) \left (\frac{\partial h}{\partial z} + 1 \right) \right]</script>
<p>where \(K\) is the hydraulic conductivity (the ease with which water can move through pore spaces or fractures), \(h\) is the potential energy of the water (matric head) induced by capillary action, \(z\) is the elevation above a vertical reference height (called datum in geodesy), \(\theta\) is the volumetric water content, and \(t\) is time.</p>
<p>We will be playing a lot with this equation in the future!</p>
<p><a href="https://kakila.bitbucket.io/posts/ground-water-recharge-rate.html">Groundwater Recharge Rate</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on June 12, 2017.</p>
<![CDATA[Polyonymous Algorithms]]>https://kakila.bitbucket.io/posts/polyonymous-algorithms2017-06-01T17:29:44+02:002017-06-01T17:29:44+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>One core idea of emulation is to represent the ensemble of outputs of a complicated
simulator in a reduced basis.
The battle-horse algorithm to do this is the famous <strong>PCA</strong>, a.k.a <strong>POD</strong>, a.k.a <strong>SVD</strong>, a.k.a <strong>EOF</strong>, a.k.a. <strong>EBD</strong>, a.k.a. <strong>KLD</strong>, a.k.a. …
<!-- more --></p>
<p>Really, this algorithm has a different name in each community that uses it!
I did not invest much time in finding whether this is due to simultaneous discoveries, or due to researchers not looking into other communities to see what already exist (i.e. re-inventing the wheel and calling it differently), or due to mischievous re-naming to get published. In case of the first there is nothing to do but wait (and hope) for unification. The second is something shouldn’t be common anymore, specially in the age of the internet. The third and last is unforgivable, it is more common than what I would like to accept, and is caused by the stupid pressure to publish <em>a lot</em> (instead of <em>good stuff</em>). I believe that the true cause is a mix of all three.</p>
<p>Anyways, I hope new researchers, with young blood, are more careful, ethical, and worry less about piling up disposable articles.</p>
<p>As I said at the beginning, one very useful approach to accelerate a complicated simulator
is to try to represent the ensemble of outputs in a reduced basis. Generally this is something of the form</p>
<script type="math/tex; mode=display">\mathbf{y}(t,\mathbf{\lambda}) = \sum_{i=1}^N \mathbf{c}_i(\mathbf{\lambda}) \varphi_i(t)</script>
<p>where \(\mathbf{y}(t,\mathbf{\lambda})\) is the output of the simulator, here assumed to depend on time and some parameter vector \(\mathbf{\lambda}\).
In the Ansatz above, we try to put the time dependence in some parameter independent basis \(\varphi_i(t)\), which is hopefully small.
The dependence on the parameters is present only in the coefficients of the basis.
Put in words, we try to represent any output of the simulator as a linear combination of a fixed basis.
In this representation, we need to learn only the parameter dependence of the basis coefficients.
We learn this dependence from outputs of the simulator obtained with different parameters, i.e.e the <em>training data</em>.
When the basis is small this can provide a boost in the calculation time of outputs at new parameter values.
The size of the basis is related to the complexity of the signals the simulator generates, more complex outputs require more basis elements to generate a good representation.</p>
<h2 id="svd-decomposition">SVD decomposition</h2>
<p>A very efficient way of obtaining the basis \(\varphi_i(t)\) is by applying the <a href="https://en.wikipedia.org/wiki/Singular_value_decomposition">Singular Value Decomposition</a> (<strong>SVD</strong>) to the so called <em>snapshots matrix</em> (or data matrix) \(\mathbb{Y}\) which contains the recorded simulation outputs in its columns</p>
<script type="math/tex; mode=display">% <![CDATA[
\mathbb{Y} = \begin{bmatrix} y(t,\mathbf{\lambda}_1) & \ldots & y(t,\mathbf{\lambda}_n)\end{bmatrix} %]]></script>
<p>This way of obtaining the basis is equivalent to:</p>
<ul>
<li>
<p>Principal Components Analysis (<strong>PCA</strong>): well know in machine learning and statistics and used for dimensionality reduction.</p>
</li>
<li>
<p>Proper Orthogonal Decomposition (<strong>POD</strong>): popular in structural mechanics, physics, and theory of differential equations.</p>
</li>
<li>
<p>Karhunen-Loève Decomposition (<strong>KLD</strong>): also well known in statistics.</p>
</li>
<li>
<p>Empirical Orthogonal Functions (<strong>EOF</strong>): popular among wind and solar power researchers.</p>
</li>
<li>
<p>…</p>
</li>
</ul>
<p>The list is really long and it gets longer the more I read from different research fields.</p>
<p>Of course there are other decomposition algorithms that are not equivalent to SVD.
For example there is <a href="https://en.wikipedia.org/wiki/Non-negative_matrix_factorization">Nonnegative Matrix Factorization</a> (<strong>NMF</strong>), <a href="https://en.wikipedia.org/wiki/Independent_component_analysis">Independent Component Analysis</a>, <a href="http://www.jmlr.org/papers/volume12/omlor11a/omlor11a.pdf">Anechoic Mixing Models</a>, my own work on kinematic chains using the <a href="http://journal.frontiersin.org/article/10.3389/fncom.2013.00191/full">Dynamic Decomposition Method</a><sup id="fnref:Carbajal12"><a href="#fn:Carbajal12" class="footnote">1</a></sup>, and the many decompositions used in <a href="https://en.wikipedia.org/wiki/Compressed_sensing">Compressed Sensing</a>.
All these other methods are also valid approaches for emulation.
They can be particularly useful if we have prior information about the basis, or if we want the basis to have a given property for the sake of interpretability of the decomposition (this is seldom required in emulation since the objective is just to accelerate the simulator and not much more).</p>
<p>A very simple example is implemented in the script <a href="https://bitbucket.org/KaKiLa/emumore/src/tip/src/mfiles/s_svd_ode.m">s_svd_ode.m</a> (available in the repository), where the motion of the pendulum is reduced to an interpolation problem. The parameter vector contains the initial angle of the pendulum and the initial speed. In this example only 4 basis elements are needed to
emulate the pendulum with an error of about 1%. you can see the surfaces of these coefficients int he plots below</p>
<figure>
<img src="https://kakila.bitbucket.io/emumore/images/s_svd_ode_fig2.png" alt="basis coefficients as function of the parameters" />
<figcaption>The 4 basis coefficients as function of the initial angle and velocity of the pendulum. Interpolation is used to emulate parameters not present in the training data.</figcaption>
</figure>
<p>We presented a concrete example of application of SVD and MNF in our publication<sup id="fnref:Carbajal16"><a href="#fn:Carbajal16" class="footnote">2</a></sup>, under the name <em>Data-driven emulation</em>.</p>
<p>Similar ideas are applied in model order reduction (we will post about the difference between model order reduction and emulation soon), you can read about them in Reduced Basis Method for PDEs<sup id="fnref:Quarteroni16"><a href="#fn:Quarteroni16" class="footnote">3</a></sup> and in POD for dynamical systems <sup id="fnref:Volkwein13"><a href="#fn:Volkwein13" class="footnote">4</a></sup>.</p>
<h1 id="references">References</h1>
<div class="footnotes">
<ol>
<li id="fn:Carbajal12">
<p>Carbajal, J. P. (2012). Harnessing Nonlinearities: Generating Behavior from Natural Dynamics. University of Zürich. https://doi.org/10.5167/uzh-66463 <a href="#fnref:Carbajal12" class="reversefootnote">↩</a></p>
</li>
<li id="fn:Carbajal16">
<p>Carbajal, J. P., Leitão, J. P., Albert, C., & Rieckermann, J. (2017). Appraisal of data-driven and mechanistic emulators of nonlinear simulators: The case of hydrodynamic urban drainage models. Environmental Modelling & Software, 92, 17–27. http://doi.org/10.1016/j.envsoft.2017.02.006. Arxiv: https://arxiv.org/abs/1609.08395 <a href="#fnref:Carbajal16" class="reversefootnote">↩</a></p>
</li>
<li id="fn:Quarteroni16">
<p>Quarteroni, A., Manzoni, A., & Negri, F. (2015). Reduced Basis Methods for Partial Differential Equations: An Introduction. Springer International Publishing. <a href="#fnref:Quarteroni16" class="reversefootnote">↩</a></p>
</li>
<li id="fn:Volkwein13">
<p>Volkwein, S. (2013). Proper orthogonal decomposition: Theory and reduced-order modelling. Lecture Notes, University of Konstanz. http://www.math.uni-konstanz.de/numerik/personen/volkwein/teaching/POD-Book.pdf <a href="#fnref:Volkwein13" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="https://kakila.bitbucket.io/posts/polyonymous-algorithms.html">Polyonymous Algorithms</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on June 01, 2017.</p>
<![CDATA[Repository goes Public!]]>https://kakila.bitbucket.io/posts/repository-goes-public2017-06-01T17:20:00+02:002017-06-01T17:20:00+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>From now on everybody is able to read and fork the content in our repostiory.
<strong>Be warned</strong>: it is in pre-alpha so mind your steps!</p>
<p><a href="https://kakila.bitbucket.io/posts/repository-goes-public.html">Repository goes Public!</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on June 01, 2017.</p>
<![CDATA[Wiki goes Public!]]>https://kakila.bitbucket.io/posts/wiki-goes-public2017-05-08T10:19:36+02:002017-05-08T10:19:36+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>From now on everybody is able to read the content in our wiki.
<strong>Be warned</strong>: the wiki content is <em>crazy</em>.</p>
<p><a href="https://kakila.bitbucket.io/posts/wiki-goes-public.html">Wiki goes Public!</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on May 08, 2017.</p>
<![CDATA[Hello world!]]>https://kakila.bitbucket.io/posts/hello-world2017-04-21T06:00:02+02:002017-04-21T06:00:02+02:00JuanPi Carbajalhttps://kakila.bitbucket.ioajuanpi@gmail.com
<p>The kickoff meeting on 19th April 2017 set the project into motion!
The official starting date is 1st of May, but we are nevertheless already setting up the activities.
<!-- more --></p>
<p>The name of the project is <strong>EmuMore</strong>, an acronym that summarizes the idea that <strong>Emu</strong>lation will allows us to do <strong>More</strong> with our models.
An emu is also a fast running bird, pure coincidence! But it gave us some ideas for a logo (it isn’t finished):</p>
<p><img src="https://kakila.bitbucket.io/emumore/images/emu.png" alt="Emumore's logo" /></p>
<p>Suggestions are welcomed.</p>
<p>Once we got the structure of the project <a href="https://bitbucket.org/kakila/emumore">source code repository</a> ready we will make it public.
Currently we are sharing just the issue tracker, which should give a nice overview of our current activities.</p>
<p>One of the main tasks is the preparation of Event #1, taking place on Nov. 2017.
But we are also preparing the first emulation datasets and benchmark problems.</p>
<p>We will keep you posted!</p>
<p><a href="https://kakila.bitbucket.io/posts/hello-world.html">Hello world!</a> was originally published by JuanPi Carbajal at <a href="https://kakila.bitbucket.io">EmuMore's blog</a> on April 21, 2017.</p>