Jekyll2019-03-13T15:10:57+08:00https://lemonteaa.github.io/(into tech :lemontea)Lemontea's tech blog: A math lover in awe of the beauty and power of high level programming languages
A Personal Update2019-03-12T22:00:00+08:002019-03-12T22:00:00+08:00https://lemonteaa.github.io/meta/2019/03/12/a-personal-update<p>Hello everyone! I haven’t updated this blog for a whole year, and I think it’s about time to give you all an update on what’s happened to me and this blog.</p>
<p>Long story short, I am not in a day job now - instead I’m in a DIY style technology retreat (sorta like the <a href="https://www.recurse.com/">recurse center</a>) where I focus on myself - to explore technologies and methods I’m curious of, to expand my horizon, to challenge and push my knowledge, capabilities, and limit, and to reflect on my career path, as well as the meaning of technologies and its relation to human progress.</p>
<p>The road has been rougher than I planned (it always <a href="https://en.wikipedia.org/wiki/Hofstadter%27s_law">is</a>, isn’t it? :wink:). As a result, I haven’t been able to spare any time for Open Source Work/Blogging, which used to be done on the side.</p>
<p>That being said, I will be doing what I can to compensate somewhat:
<!--more--></p>
<ul>
<li>In the spirit of Open Source, I am opening up <a href="/drafts/">a new section</a> on this blog - it contains a collection of writings/works that are not <em>production-ready</em> yet, but which nonetheless may be already useful, in raw form, to some people.</li>
<li>I will be shifting my overall direction to <em>more execution</em> (over introspection/learning) in this phase of the retreat, given that I’ve done a round of catching up with basic knowledge already. This will hopefully make some of the planned blog posts ready faster. (I will explain this point below)</li>
</ul>
<p>When I just began this blog, my vision is for it <strong>to be a place where I promote/advocate for relatively advanced software development technologies (interpreted in a broad way) in a forward-thinking, but still pragmatic manner</strong>. Because of this I cannot just write any posts here: if I pick an easy topic, it will not be aligned with this vision and risks diluting/detracting from the main goal. On the other hand, for those topics that are aligned, they are either so beyond my comfort zone that I cannot honestly find anything substantial and valuable to say; or they involve hands-on projects that I haven’t myself completed yet. So I’m in a bind - in order to be able to write a post, I have to either 1) learn something that’s pretty hard relative to what I already know (I’m gradually getting there), or 2) do some projects that’re just beyond the extent of my ability. (I’m trying - just not done yet)</p>
<p>Now that the main business is dealt with, below are some relatively tangent topics. Read at your leisure :grimacing:</p>
<h2 id="rationale">Rationale</h2>
<p>Why pursue <em>advanced</em> software development? This is not as trivial a question as may be perceived by some geeks. The context is a rather unspoken cultural divide (compare: <a href="https://en.wikipedia.org/wiki/The_Two_Cultures">the two cultures</a> of science vs humanities) in the software industry:</p>
<ul>
<li>On one side is the business culture, where economy reigns supreme and both engineering and technologies are seen as just a tool (among many others) to some ends, being subservient to business.</li>
<li>On the other side is the <a href="https://en.wikipedia.org/wiki/Hacker_culture">hacker culture</a>, where technologies are seen as a potent, powerful force, that when correctly used can further human potentials by opening up new possibilities, especially those that are forbidden explicitly or implicitly by authorities and society. <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup></li>
</ul>
<p>If you’re on the Hacker’s camp, using advanced technique to push the state of the art of software development forward would be a no-brainer. What they may have missed is that they may think everyone think this is a no-brainer, when in fact this is not true. From the eyes of someone who live and speak business, this is far from being <em>a priori</em> true - they are justified only to the extent that it increases profit, and after accounting for the costs and risks involved in this path, <strong>including the opportunity costs of pursuing other paths instead</strong>.</p>
<p>So, does that mean I’m biased? That I’m just a naive geek? (I hope not, but hey, it’s not productive to live in constant self-doubt) I still believe that there is unused potential in advanced method.</p>
<p>Software Engineering lives in quiet desparation of <em>just not working</em>. <sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> Despite decades of effort on almost every angle of research people can come up with, our industry, as a whole, still cannot make software consistently and reliably. Schedule delay, costly bugs, over-budget, all these nightmares are a fact of life, and life in this state is akin to what it is like before modern medicine - we can try a cure, but there is no guarantee.</p>
<p>What can we do then? Can this situation be improved? Some people believe that ultimately it is the <a href="https://www.amazon.com/Peopleware-Productive-Projects-Teams-3rd/dp/0321934113">human factor</a> that limits what we can solve and that there is <a href="https://www.amazon.com/Mythical-Man-Month-Software-Engineering-Anniversary/dp/0201835959">no universal formula</a>, no magic cure. While I respect the importance of these factors, I also believe that an attitude of disregarding, or dismissing, technical factor as trivial is counter-productive - most of us are not operating at a range where adding/improving software development technologies give diminishing marginal returns. <strong>Unfortunately, large returns may require global optimization - fundamentally changing the way we do development - instead of local, incremental improvements</strong>.</p>
<p>In short, we do not yet have a <em>science</em> of software engineering, and hence the appropiate action to take in the long run is to explore, formulate, and test hypothesis, and to be especially aware of any potential pre-conceptions that may be completely wrong.</p>
<h2 id="t-model-of-skills-and-my-project-list">T-model of skills and my project list</h2>
<p>The T-model of skills posits that a good engineer should have a broad understanding of various disciplines at a basic level (the horizontal bar), with one (or more) specialty that he drills down into and possess deep, expert skills (the vertical stroke). My own target list with respect to softwares is as follows:</p>
<ul>
<li>AI - Mainly neural network and reinforcement learning</li>
<li>DevOps - Making operation less of a black art</li>
<li>Software Architecture and Design - life after MVC</li>
<li>Modern Frontend web development - Single Page Application (SPA) Frameworks, and recent features in browsers (also, HTML5)</li>
<li>Computer Graphics</li>
</ul>
<p>My background’s mainly in backend, and that’s where I’m going deep into. To that end I have planned a number of exercises for myself:</p>
<ul>
<li>Build a Big Data Pipeline using: Onyx, Kafka, some NoSQL DB (Cassandra?)</li>
<li>Realtime live chat/videa streaming type app using: <a href="https://github.com/ptaoussanis/sente">sente</a>, WebRTC etc</li>
<li>Complex Domain Modelling with Datomic</li>
<li>Microservice/API using: Lacina (for GraphQL), <a href="https://github.com/juxt/yada">yada</a>?</li>
</ul>
<h2 id="closing">Closing</h2>
<p>Thanks for bearing with me. May the source be with you!</p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:1">
<p>This does <em>not</em> mean that hackers are anti-social. It simply means that they prefer to judge the morality of things more on whether they are actually of benefits, than by the dictate of social norms <em>in and of itself</em>. See also: <a href="https://en.wikipedia.org/wiki/Social_constructionism">Social Constructionism</a>, <a href="https://en.wikipedia.org/wiki/Critical_theory">Critical Theory</a> <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>This is an open secret, and a kind of “family shame”. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>Hello everyone! I haven’t updated this blog for a whole year, and I think it’s about time to give you all an update on what’s happened to me and this blog. Long story short, I am not in a day job now - instead I’m in a DIY style technology retreat (sorta like the recurse center) where I focus on myself - to explore technologies and methods I’m curious of, to expand my horizon, to challenge and push my knowledge, capabilities, and limit, and to reflect on my career path, as well as the meaning of technologies and its relation to human progress. The road has been rougher than I planned (it always is, isn’t it? :wink:). As a result, I haven’t been able to spare any time for Open Source Work/Blogging, which used to be done on the side. That being said, I will be doing what I can to compensate somewhat:How to get into the brave new world of Neural Network and Reinforcement Learning2018-02-08T22:00:00+08:002018-02-08T22:00:00+08:00https://lemonteaa.github.io/misc/2018/02/08/how-to-get-into-the-brave-new-world-of-neural-network-and-reinforcement-learning<p><em>(Note: This is a short preparatory post for the real one, where I plan to share my experience dabbling with the rusty but beautiful <a href="https://github.com/matthiasplappert/keras-rl">keras-rl</a> library. I will release that after I finally fulfill my promise and deliver the PPO+A2C integration that has been dragged longer than I am comfortable with :stuck_out_tongue:)</em></p>
<p><em>(Hint: some of the not-explicitly-referenced links are just Wiki, but many others are hidden treasure troves. Thanks for all these authors for their amazing works! :blush:)</em></p>
<p>Modern Artificial Intelligence is a paradoxical case of an unstoppable force meeting an unmovable object. On one hand, this field has been so hyped in the last few years that some people worry about a bubble and potential backlash (probably bad memories from the last <a href="https://en.wikipedia.org/wiki/AI_winter">AI winter</a> ?); on the other hand its progress is undeniable even to hardcore skeptics - even if we assume that there will be no more new breakthrough (which is quite unlikely), fully exploiting/applying the technology available right now to its potential will still take a long long time (if nothing else, because the list of new applications opening up from this round of new methods is huuuge :wink: ). This is an interesting time indeed.</p>
<p>So, if you’re here, you probably wants to get into the field, but just like any bandwagon, the signal-to-noise ratio isn’t exactly nice. How do I gain a foothold and stay?
<!--more--></p>
<p>I personally believe that principles matter. To borrow some armchair buddhism, things in this world are impermanent, always in flux - whatever libraries, frameworks, or even language you hold dear has no guarantee to stay alive, much less the same, as the industry moves forward (or in circle?) But ideas live on, and may even reincarnate when the time is right.</p>
<p>Which is why I’m now outlining a list of the enduring “basics”, to answer this question.</p>
<p>To make things simple, it boils down to these few things:</p>
<h1 id="fundamental">Fundamental</h1>
<ul>
<li>Learn programming, preferably Python (because of its strong ecosystem, mainstream status, and beginner-friendliness)</li>
<li>Learn some scientific computing (the usual barrage of <a href="https://pandas.pydata.org/">panda</a>, <a href="https://matplotlib.org/">matplotlib</a>, <a href="http://www.numpy.org/">numpy</a> etc for Python), specifically functions/syntax for working with vector/matrix/tensor <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> - i.e. <a href="https://stackoverflow.com/questions/1422149/what-is-vectorization">vectorization</a> (See <a href="https://www.mathworks.com/help/matlab/matlab_prog/vectorization.html?s_tid=gn_loc_drop">this</a> for the guide in matlab), and also try out some <a href="https://www.wolframalpha.com/input/?i=derivative+of+x%5E4+sin+x&lk=3">symbolic computing</a> (The Wolfram code for that is <code class="highlighter-rouge">f[x_] := x^4 * Sin[x]</code> then <code class="highlighter-rouge">f'[x]</code>, see the <a href="http://reference.wolfram.com/language/ref/Derivative.html">doc</a>)</li>
<li>Learn math (good if undergrad major level in both pure and applied maths, best if PhD in selected areas :smirk: )
<ul>
<li>Linear algebra (the neural interconnections are expressed by matrix, and <a href="http://colah.github.io/posts/2014-07-Understanding-Convolutions/">convolution</a> is an important example of a <a href="https://en.wikipedia.org/wiki/Bounded_operator">linear operator</a> <sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>)</li>
<li>Calculus (to help do manual computation related to the quantified performance of a network)</li>
<li>Probability theory/Statistic/<a href="https://en.wikipedia.org/wiki/Stochastic_process">Stochastic process</a> (The main trend of modern AI is statistical in nature<sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup> - Instead of analyzing any particular training/testing data, we treat the whole ensemble/possible inputs seem in real life statistically, which is much more tractable. Stochastic process, which is roughly speaking the more eloquent brother of probability theory, is useful for more complicated setup like in reinforcement learning, where the basic framework is already <a href="https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/markov_decision_process.html">Markov Decision Process</a>)</li>
<li>Optimization algorithm/theory (The whole Neural Network thing is just optimizing a function’s fit to data by tuning parameters to minimize error while constraining the complexity of our function, to be deliberately provocative. Anyway, the first algorithm you will see is the <a href="http://ruder.io/optimizing-gradient-descent/">Stochastic Gradient Descent</a>, which comes from the simple and naive <a href="https://distill.pub/2017/momentum/">gradient descent</a>, but have some not-so-obvious new behavior.)</li>
</ul>
</li>
</ul>
<h1 id="concepts-in-machine-learning">Concepts in Machine Learning</h1>
<p>Once you’re done with that, or at least have a good enough understanding, go and learn basic concepts in machine learning:</p>
<ul>
<li>The data-centric, or statistical paradigm,</li>
<li>supervised vs (un/semi)-supervised learning,</li>
<li>the basic workflow,</li>
<li>feature engineering,</li>
<li>choosing algorithm/methods,</li>
<li>overfitting/generalization problem,</li>
<li>pitfalls in <a href="https://xkcd.com/882/">testing</a> <sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup> /cross validation</li>
</ul>
<p>To list a few.</p>
<p>After that, it is really no different from any other learning:</p>
<p>Search, read, think/tinker, repeat</p>
<p>The end. (See? It’s not that hard afterall :laughing:)</p>
<p>Actually, not yet… to help you along (and to thank you for bearing with me for a supposedly “short note”), here is a filtered list of links/resources I find helpful:</p>
<h1 id="references">References</h1>
<p>For a comprehensive introduction to all these diverse ideas, the <a href="https://www.gitbook.com/book/leonardoaraujosantos/artificial-inteligence/details">e-book written by Leonardo Araujo dos Santos</a> seems to do the tricks.</p>
<p>For understanding neural network, check out this <a href="http://neuralnetworksanddeeplearning.com/">book</a>. It’s a bit long, but very worth it as a long term investment. If you’re short on time/prefer a more hands-on approach, then see <a href="http://ufldl.stanford.edu/tutorial/">Stanford’s basic tutorial</a>.</p>
<p>We will be skipping all those baroque network architecture as that’s not our focus today.</p>
<p>For reinforcement learning, the resources are scattered:</p>
<ul>
<li>
<p><a href="http://karpathy.github.io/2016/05/31/rl/">Andrej Karpathy’s blog post</a> is a popular introductory post that’s nonetheless deep in some aspects.</p>
</li>
<li>
<p><a href="https://rubenfiszel.github.io/posts/rl4j/2016-08-24-Reinforcement-Learning-and-DQN.html">Ruben Fiszel’s blog post</a> - He is an intern who learnt massively in that short timespan and as a result has a trove of treasure/insights/experience to share</p>
</li>
</ul>
<p>More formally, there are some tutorial series that gradually ease in people to successively more complex concepts: <a href="https://medium.com/emergent-future/simple-reinforcement-learning-with-tensorflow-part-0-q-learning-with-tables-and-neural-networks-d195264329d0">this</a>, and <a href="https://medium.com/machine-learning-for-humans/reinforcement-learning-6eacf258b265">this</a>.</p>
<p>In short, learn about the setup of reinforcement learning, then learn about the theory of MDP, then learn about Q-function/value-based approach, then policy-based method, then the actor-critic architecture.</p>
<p>Congratulations, you’re now ready! :clap:</p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Please note that while for computational purpose a vector can be treated as a 1 by n (or n by 1) matrix and a matrix can be treated as a second rank tensor, in math/physics they are not exactly the same thing. See <a href="https://physics.stackexchange.com/questions/20437/are-matrices-and-second-rank-tensors-the-same-thing">this</a> for a physics point of view and <a href="https://math.stackexchange.com/questions/412423/differences-between-a-matrix-and-a-tensor">this</a> for a math point of view. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Well I cheated a bit, since there is both a discrete and continuous version of convolution, and for the continuous version you’ve got <a href="https://en.wikipedia.org/wiki/Mathematical_analysis">analytic</a> issues such as <a href="https://math.stackexchange.com/questions/172504/why-do-we-say-the-harmonic-series-is-divergent">convergence</a> or boundedness, etc. I’m not sure whether the <a href="https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space">application of functional analysis in machine learning</a> is relevant at an introductory level though. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>Two remarks here. First is that if you want to go <em>really</em> hardcore, see the topic called “Statistical Learning Theory”. It isn’t exactly necessary if your goal is to merely understanding enough theory to be able to work effectively with those tools in practise though. Second point is that in the last <del>hype cycle</del> generation, the main trend of AI is to use <a href="https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence">symbolic method</a>. Given the massive success story of AI recently, this predictably spaked a debate on the future of AI, especially since <a href="https://en.wikipedia.org/wiki/Technological_singularity">Singularitarians</a> such as those in the <a href="https://intelligence.org/">MIRI</a> or <a href="http://lesswrong.com/">Less Wrong</a> believes that strong/general AI is not only possible, but could arrive pretty soon (and once you have that, it can then bootstrap/improve itself to become super-intelligent in short order). In particular, is statistical method really superior to symbolic method? See <a href="https://www.tor.com/2011/06/21/norvig-vs-chomsky-and-the-fight-for-the-future-of-ai/">this</a> for an example debate. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>Jokes and obligatory xkcd reference aside, data dredging/<a href="https://www.wired.com/2013/02/big-data-means-big-errors-people/">spurious correlation</a> is indeed a serious problem even in traditional statistics. Cross validation is one standard technique to avoid such trap. Here, the new problem is to test using the same data set you trained the program/model with. (Which is the generalization problem in the last bullet point I think?) <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>(Note: This is a short preparatory post for the real one, where I plan to share my experience dabbling with the rusty but beautiful keras-rl library. I will release that after I finally fulfill my promise and deliver the PPO+A2C integration that has been dragged longer than I am comfortable with :stuck_out_tongue:) (Hint: some of the not-explicitly-referenced links are just Wiki, but many others are hidden treasure troves. Thanks for all these authors for their amazing works! :blush:) Modern Artificial Intelligence is a paradoxical case of an unstoppable force meeting an unmovable object. On one hand, this field has been so hyped in the last few years that some people worry about a bubble and potential backlash (probably bad memories from the last AI winter ?); on the other hand its progress is undeniable even to hardcore skeptics - even if we assume that there will be no more new breakthrough (which is quite unlikely), fully exploiting/applying the technology available right now to its potential will still take a long long time (if nothing else, because the list of new applications opening up from this round of new methods is huuuge :wink: ). This is an interesting time indeed. So, if you’re here, you probably wants to get into the field, but just like any bandwagon, the signal-to-noise ratio isn’t exactly nice. How do I gain a foothold and stay?Getting Started With Clojurescript Development - Setup2017-10-11T22:35:00+08:002017-10-11T22:35:00+08:00https://lemonteaa.github.io/tutorials/2017/10/11/getting-started-with-clojurescript-development-setup<p>We have seen miraculous progress in frontend development in these few years - the domination/monopoly of javascript, AJAX, node.js, the rise and rapid iteration of javascript libraries/frameworks and Single Page Application (with the current crop being Angular, React, Vue, as well as many others). But we’ve also experienced growing pain - the churn rate of libraries is way too high<sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>, javascript as a language (and as part of the toolchain) is too hopelessly ill-equipped<sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> to deal with the demand of modern frontend space, with attendant attempts to either patch over (transpiling languages such as Coffeescript and Typescript), reform (<a href="https://benmccormick.org/2015/09/14/es5-es6-es2016-es-next-whats-going-on-with-javascript-versioning/">ES-whatever</a>), or outright replace (<a href="http://webassembly.org/">WebAssembly</a>) it.</p>
<p><a href="https://clojurescript.org/">Clojurescript</a> is a solution in the “patch-over” category that bring the elegance and pleasure of writing program in Clojure to the web. More importantly, Clojurescript has placed a strong emphasis on Interactive development, and has been leading other technology stack over the years.
<!--more--></p>
<h2 id="background">Background</h2>
<p>This section is written to help frontend developer / non-programmer get some context so they can understand where some of the apparently strange things in Clojurescript tooling come from. If you are a backend developer, please feel free to skip this section. (By the way, I’m not an expert in tooling so if you have deeper info, please leave a comment below. Thanks!)</p>
<p>Before Clojurescript, there is Clojure, a JVM-hosted, general purpose language. JVM is a nice platform that largely eliminates cross-platform issues (barring the JNI and JNA) while providing competitive performance (except bare-metal), and has widespread adoption especially in the enterprise world. The standard language on top of JVM is Java (not to be confused with javascript of course), which is arguably flawed, but has been used and deployed in so many places that it is effectively irreplaceable. Clojure is a language built on top of Java, and so can be run on the JVM. This choice let us leverage both the strength of the JVM platform, as well as the monumental amount of libraries in Java.</p>
<p>The major build tools for Clojure (and Clojurescript as it turns out) is lein and boot. A simple way to distinguish them is that lein is declarative while boot is more imperative. I will not go into discussing their relative merit and will use lein all the way. (Ya, that’s lazy, I know)</p>
<p>So what’s the point of all these talks? Although Clojurescript compiles to javascript, it still uses lein/boot as its build tool (which make sense in retrospect - it is really bad to fragment something as fundamental as build tool unnecessarily - and Clojure/Clojurescript is supposed to be very close, differing only in their interpretive/compilation target), and so you will need to install the JDK/JRE/JVM (See this <a href="https://www.javatpoint.com/difference-between-jdk-jre-and-jvm">page</a> for the fine prints of their differences), as lein is a Clojure program and so ultimately runs on that.</p>
<h2 id="prerequisite">Prerequisite</h2>
<p>As the last section explains, you will need to have installed the JDK (Java Development Kit - which includes an implementation of JVM plus development tools such as the Java Compiler.)/JVM and then lein.</p>
<p>For the JDK, it is recommended to use JDK 8 though version 6 or 7 works as well - see the <a href="https://clojure.org/guides/getting_started">official website</a>. One caveat - there was some compatibility issue with JDK 9 - see <a href="https://www.reddit.com/r/Clojure/comments/71p5mv/java_9_has_been_released/">here</a> for example.</p>
<p>Once you have a working Java installation, visit the homepage of <a href="https://leiningen.org/">lein</a> and follow the install instruction there.</p>
<h2 id="a-basic-project-using-figwheel">A Basic Project Using Figwheel</h2>
<p>After getting the prerequisites, let’s get started! Much like other modern ecosystems, this particular step has been “optimized” and is as easy as opening up your command prompt and typing:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>lein new figwheel hello-world -- --reagent
</code></pre>
</div>
<p>Which will create a new folder hello-world containing a barebone project. (In frontend we call this kind of project-generation-from-template scaffolding) <a href="https://github.com/bhauman/lein-figwheel">Figwheel</a> is the tool that provides all those interactive experiences and niceties (we’ll see more later), while the last argument specifies which Clojurescript web framework to use (We choose reagent here).</p>
<p>For your reference here is the console output when generating new project:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>Retrieving figwheel/lein-template/0.5.13/lein-template-0.5.13.pom from clojars
Retrieving figwheel/lein-template/0.5.13/lein-template-0.5.13.jar from clojars
Generating fresh 'lein new' figwheel project.
Change into your 'hello-world' directory and run 'lein figwheel'
Wait for it to finish compiling
Then open 'http://localhost:3449/index.html' in your browser
</code></pre>
</div>
<p>To run the project, just cd into the folder and type:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>lein figwheel
</code></pre>
</div>
<p>Figwheel will pop up a browser tab (with default location <code class="highlighter-rouge">localhost:3449</code>) connected to your project, and the command prompt will turn into a Clojurescript prompt/REPL.</p>
<p>Here is a screenshot of what you get in the browser:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Basic_1_Reagent.png" title="ClojureScript Barebone Project (Reagent)" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Basic_1_Reagent.png" alt="ClojureScript Barebone Project (Reagent)" />
</a>
<p class="image-caption">A barebone Clojurescript Project using the Reagent framework plus figwheel</p>
</div>
<p>while here is the console: (Notice that the first time you run a project, it will automatically download and install dependencies which may take a while. I’ve hidden those extra log for clarity)</p>
<div class="highlighter-rouge"><pre class="highlight"><code>*** Warning: This project requires Leiningen 2.7.1, but you have 2.6.1 ***
Get the latest version of Leiningen at http://leiningen.org or by executing
"lein upgrade".
Figwheel: Cutting some fruit, just a sec ...
Retrieving reagent/reagent/0.7.0/reagent-0.7.0.pom from clojars
Retrieving org/clojure/clojurescript/1.9.655/clojurescript-1.9.655.pom from central
Retrieving com/google/javascript/closure-compiler-unshaded/v20170521/closure-compiler-unshaded-v20170521.pom from central
...(snipped)...
Figwheel: Validating the configuration found in project.clj
Figwheel: Configuration Valid ;)
Figwheel: Starting server at http://0.0.0.0:3449
Figwheel: Watching build - dev
Figwheel: Cleaning build - dev
Compiling "resources/public/js/compiled/hello_world.js" from ["src"]...
Successfully compiled "resources/public/js/compiled/hello_world.js" in 27.267 seconds.
Figwheel: Starting CSS Watcher for paths ["resources/public/css"]
Launching ClojureScript REPL for build: dev
Figwheel Controls:
(stop-autobuild) ;; stops Figwheel autobuilder
(start-autobuild [id ...]) ;; starts autobuilder focused on optional ids
(switch-to-build id ...) ;; switches autobuilder to different build
(reset-autobuild) ;; stops, cleans, and starts autobuilder
(reload-config) ;; reloads build config and resets autobuild
(build-once [id ...]) ;; builds source one time
(clean-builds [id ..]) ;; deletes compiled cljs target files
(print-config [id ...]) ;; prints out build configurations
(fig-status) ;; displays current state of system
(figwheel.client/set-autoload false) ;; will turn autoloading off
(figwheel.client/set-repl-pprint false) ;; will turn pretty printing off
Switch REPL build focus:
:cljs/quit ;; allows you to switch REPL to another build
Docs: (doc function-name-here)
Exit: Control+C or :cljs/quit
Results: Stored in vars *1, *2, *3, *e holds last exception object
Prompt will show when Figwheel connects to your application
To quit, type: :cljs/quit
dev:cljs.user=>
</code></pre>
</div>
<p>Before we move on, let me try to explain what’s happening behind the scene (This is quite complicated and I am really not sure I got all the details right, so yup - proceed at your own risk).</p>
<p>So we started the build tool/task runner lein. Figwheel (running as an ordinary program in your OS with file system access) will start to watch your source directory for any file change and invoke recompilation in real time. This is known as auto-rebuilding, and is actually provided by another plugin <a href="https://github.com/emezeske/lein-cljsbuild">Cljsbuild</a> which figwheel wraps.</p>
<p>At the same time it also launches a REPL environment that’s also provided in a vanilla clojurescript build. Since clojurescript compiles to javascript, it obviously needs a javascript execution environment. This makes thing more complicated as we want REPL to run in the build tool/lein we just spun up - we need to somehow connect them.</p>
<p>It turns out there is more than the obvious execution environment (browser vs node.js vs others, in case you’re long out-of-sync with the frontend space). For the case of browser, we just need to go to the browser (why, my dear ;) ) and open the page that loads those compiled scripts. It ought to also contain additional code fragment that makes a web socket connection to the lein process and then add the “suitable” event handler. (Web socket is used because it is two-way and real time, besides, it’s sorta cool)</p>
<p>Back in the REPL, it does listen for such connections and once they’re good, we can enter forms to evaluate, which the REPL handles by sending it to the browser/execution environment through the connection. The running program in the browser handle that message by calling clojurescript’s own eval inside its own environment, and then sending the result back.</p>
<p>So what additional values does figwheel provide? The answer is auto-reloading: plain clojurescript + REPL only re-compiles them. To do this, the “event handler” on browser’s side is <a href="https://github.com/bhauman/lein-figwheel/blob/master/support/src/figwheel/client/file_reloading.cljs">enhanced</a> to also accept command to reload particular namespace, which it do by default through calling <a href="https://google.github.io/closure-library/api/goog.net.jsloader.html">functions</a> in the Google Closure Library; the REPL is also beefed up to send those commands. Moreover, figwheel is smart enough to figure out what files truly need reloading and only reload those.</p>
<p>Another, slightly silly answer is that figwheel do the “go to browser and open the webpage” part for you automatically :P. It also hosts its own web server to serve those compile javascript and html files.</p>
<h2 id="a-slow-tour-of-what-we-can-do">A Slow Tour of what we can do</h2>
<h3 id="live-code-reloading-and-live-repl">Live code reloading and live REPL</h3>
<p>Much like other web ecosystems, we can edit our code, save, and see the changes reflected immediately in browser. The main differences in figwheel are: <sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup></p>
<ul>
<li>No actual page reload is triggered</li>
<li>Application state is preserved (provided you write reloadable code)</li>
</ul>
<p>So let’s try it in the context of interactively developing a non-trivial application. Go to the file <code class="highlighter-rouge">src/hello_world/core.cljs</code> and edit as below, and then save:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="nf">ns</span><span class="w"> </span><span class="n">hello-world.core</span><span class="w">
</span><span class="p">(</span><span class="no">:require</span><span class="w"> </span><span class="p">[</span><span class="n">reagent.core</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">reagent</span><span class="w"> </span><span class="no">:refer</span><span class="w"> </span><span class="p">[</span><span class="n">atom</span><span class="p">]]</span><span class="w">
</span><span class="p">[</span><span class="n">clojure.set</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">s</span><span class="p">]))</span><span class="w">
</span><span class="c1">;;...
</span><span class="p">(</span><span class="k">def</span><span class="w"> </span><span class="n">rainbow</span><span class="w">
</span><span class="p">[</span><span class="s">"#FF0000"</span><span class="w">
</span><span class="s">"#FF7F00"</span><span class="w">
</span><span class="s">"#FFFF00"</span><span class="w">
</span><span class="s">"#00FF00"</span><span class="w">
</span><span class="s">"#0000FF"</span><span class="w">
</span><span class="s">"#4B0082"</span><span class="w">
</span><span class="s">"#9400D3"</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">gen-style</span><span class="w"> </span><span class="p">[</span><span class="n">n</span><span class="w"> </span><span class="n">k</span><span class="p">]</span><span class="w">
</span><span class="p">{</span><span class="no">:width</span><span class="w"> </span><span class="p">(</span><span class="nb">*</span><span class="w"> </span><span class="p">(</span><span class="nb">/</span><span class="w"> </span><span class="n">k</span><span class="w"> </span><span class="n">n</span><span class="p">)</span><span class="w"> </span><span class="mi">100</span><span class="p">)</span><span class="w">
</span><span class="no">:height</span><span class="w"> </span><span class="mi">12</span><span class="w">
</span><span class="no">:background-color</span><span class="w"> </span><span class="p">(</span><span class="nb">get</span><span class="w"> </span><span class="n">rainbow</span><span class="w"> </span><span class="p">(</span><span class="nf">mod</span><span class="w"> </span><span class="p">(</span><span class="nb">-</span><span class="w"> </span><span class="n">k</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="mi">7</span><span class="p">))</span><span class="w">
</span><span class="no">:margin</span><span class="w"> </span><span class="s">"auto"</span><span class="p">})</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">hanoi-tower</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">hanoi-rod</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">pos</span><span class="w"> </span><span class="n">discs</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:display</span><span class="w"> </span><span class="s">"inline-block"</span><span class="w"> </span><span class="no">:width</span><span class="w"> </span><span class="mi">120</span><span class="p">}}</span><span class="w">
</span><span class="p">(</span><span class="nb">map</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">k</span><span class="p">]</span><span class="w"> </span><span class="p">[</span><span class="no">:div</span><span class="w"> </span><span class="p">{</span><span class="no">:style</span><span class="w"> </span><span class="p">(</span><span class="nf">gen-style</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="n">k</span><span class="p">)}])</span><span class="w"> </span><span class="n">discs</span><span class="p">)])]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-rod</span><span class="w"> </span><span class="s">"left"</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">4</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-rod</span><span class="w"> </span><span class="s">"middle"</span><span class="w"> </span><span class="p">[])</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-rod</span><span class="w"> </span><span class="s">"right"</span><span class="w"> </span><span class="p">[])]))</span><span class="w">
</span><span class="c1">;;
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">hello-world</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">[</span><span class="no">:h1</span><span class="w"> </span><span class="p">(</span><span class="no">:text</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="p">)]</span><span class="w">
</span><span class="p">[</span><span class="no">:h3</span><span class="w"> </span><span class="s">"Edit this and watch it change!"</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-tower</span><span class="p">)])</span><span class="w">
</span><span class="c1">;;
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">find-aux</span><span class="w"> </span><span class="p">[</span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">x</span><span class="w"> </span><span class="p">(</span><span class="nf">s/difference</span><span class="w"> </span><span class="o">#</span><span class="p">{</span><span class="no">:left</span><span class="w"> </span><span class="no">:middle</span><span class="w"> </span><span class="no">:right</span><span class="p">}</span><span class="w"> </span><span class="o">#</span><span class="p">{</span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">})]</span><span class="w">
</span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">=</span><span class="w"> </span><span class="p">(</span><span class="nb">count</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">first</span><span class="w"> </span><span class="n">x</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">throw</span><span class="w"> </span><span class="p">(</span><span class="nf">js/Error.</span><span class="w"> </span><span class="p">(</span><span class="nb">pr-str</span><span class="w"> </span><span class="s">"Can't find aux: "</span><span class="w"> </span><span class="n">x</span><span class="p">))))))</span><span class="w">
</span></code></pre>
</div>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Basic_2_Reloading_A.png" title="Live reloading" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Basic_2_Reloading_A.png" alt="Live reloading" />
</a>
<p class="image-caption">Live reloading without page refresh. Note the HUD at bottom left.</p>
</div>
<p>Notice the changes in both browser tab as well as browser console output (a line is printed that shows which files are reloaded), and the fact that there is no page refresh. Also notice the flashing Clojurescript logo at bottom left. This is called the Head-up-Display (HUD) (More on this later). Going back to our application, we now have a very basic Hanoi Tower UI - which is totally hacked together - going.</p>
<p>The console we launched our application with is now a live REPL with direct access to the currently running application. We can try this out by invoking functions we’ve just entered, after application launch:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dev:cljs.user=> (require '[hello-world.core :as c])
nil
dev:cljs.user=> (c/find-aux :left :right)
:middle
dev:cljs.user=> (c/find-aux :left 4)
#object[Error Error: "Can't find aux: " #{:right :middle}]
hello_world$core$find_aux (file:/C:/Users/Owner/Documents/clojure_exp/figwheel-basic/hello-world/src/hello_world/core.cljs:57:11)
nil
dev:cljs.user=>
</code></pre>
</div>
<p>We then proceed to write a recursive function that solves the puzzle by returning a sequence of moves (each move is a vector of the form <code class="highlighter-rouge">[from to]</code>), right inside the REPL:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dev:cljs.user=>
(defn hanoi-solution [n src dst]
(if (= n 1)
[[src dst]]
(let [aux (c/find-aux src dst)
pre (hanoi-solution (- n 1) src aux)
post (hanoi-solution (- n 1) aux dst)]
(concat pre [[src dst]] post))))
#'cljs.user/hanoi-solution
</code></pre>
</div>
<p>And tests in a similar way:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dev:cljs.user=> (hanoi-solution 4 :left :right)
([:left :middle]
[:left :right]
[:middle :right]
[:left :middle]
[:right :left]
[:right :middle]
[:left :middle]
[:left :right]
[:middle :right]
[:middle :left]
[:right :left]
[:middle :right]
[:left :middle]
[:left :right]
[:middle :right])
dev:cljs.user=>
</code></pre>
</div>
<p>Now that it works, copy the function definition and save the file.</p>
<p>Let’s go back to working on the UI. Right now the discs are hard-coded, so let’s modify our state and starts wiring them in.</p>
<p>Edit the file with the following structure (Note that the <a href="https://clojure.org/reference/atoms">atom</a> here is an reagent atom - it is watched by reagent to trigger UI update whenever its content is changed):</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="nf">defonce</span><span class="w"> </span><span class="n">app-state</span><span class="w"> </span><span class="p">(</span><span class="nf">atom</span><span class="w"> </span><span class="p">{</span><span class="no">:text</span><span class="w"> </span><span class="s">"Hello world!"</span><span class="w"> </span><span class="no">:nb-disc</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="no">:disc</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="no">:left</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">4</span><span class="p">]</span><span class="w"> </span><span class="no">:middle</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="no">:right</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}))</span><span class="w">
</span></code></pre>
</div>
<p>Save. Then try to extract value at the REPL:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dev:cljs.user=> (:nb-disc @c/app-state)
nil
</code></pre>
</div>
<p>Wait, it didn’t work! Why?</p>
<p>At this point, we’ll need to discuss what “reloadable code” mean. Literally, they are simply code that are safe to be reloaded. To a first order approximation to be reloadable is to be purely functional - calling code that define a pure function twice will merely redefine it, in effect rebinding it to the same symbol (More accurately the <a href="https://clojure.org/reference/vars">Vars</a> named by the <a href="https://stackoverflow.com/questions/11662084/why-does-clojure-distinguish-between-symbols-and-vars">symbol</a> in Clojure(script) to avoid confusion). Therefore one can already get pretty far in having reloadable code by following the recommended style of writing Clojure(script) using mostly pure functions.</p>
<p>Of course, real application cannot be completely pure - they by necessity will have some state. However one can still manage them by, say adopting the <a href="https://8thlight.com/blog/uncle-bob/2012/08/13/the-clean-architecture.html">Onion architecture</a> etc. Regardless of the name, the basic idea here is to be explicit about your states and push it to the “edge” of the system (if you’re doing backend) or to have a central store of states and channel all states updates through a common interface (Which is part of the core idea of the <a href="https://medium.com/@cabot_solutions/flux-the-react-js-application-architecture-a-comprehensive-study-fd2585d06483">flux architecture</a>). Explicit states can be made to be reloadable through <code class="highlighter-rouge">defonce</code> - this ensure it won’t be redefined (and hence overwritten) if it is already defined.</p>
<p>So to recap, our program should be written to consists of just pure functions plus an explicit application state, and we want that state to survive a reload, and this is done through using <code class="highlighter-rouge">defonce</code>. The cost to pay is that whenever we change our schema for the application state, we would need to “migrate” that schema manually.</p>
<p>In theory, following the instructions above should mostly suffices. In practise though things are complicated and one needs to be careful not to accidentally introduce (hidden) mutable states, and these can be pretty subtle. For this particular point <a href="https://practicalli.github.io/clojurescript/figwheel/re-loadable-code.html">this page</a> explains it much better than I can hope to, so be sure to check it out.</p>
<p>No big deal. Back at the REPL:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dev:cljs.user=> (swap! c/app-state assoc :nb-disc 4)
{:text "Hello world!", :nb-disc 4}
dev:cljs.user=> (swap! c/app-state assoc :disc { :left [1 2 3 4] :middle [] :right [] })
{:text "Hello world!",
:nb-disc 4,
:disc {:left [1 2 3 4], :middle [], :right []}}
dev:cljs.user=>
</code></pre>
</div>
<p>Now we’re back in the game/business of updating the UI. Edit the source file and save:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">hanoi-tower</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">hanoi-rod</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">pos</span><span class="w"> </span><span class="n">discs</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="n">pos</span><span class="w"> </span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:display</span><span class="w"> </span><span class="s">"inline-block"</span><span class="w"> </span><span class="no">:width</span><span class="w"> </span><span class="mi">120</span><span class="p">}}</span><span class="w">
</span><span class="p">(</span><span class="nb">map</span><span class="w"> </span><span class="p">(</span><span class="k">fn</span><span class="w"> </span><span class="p">[</span><span class="n">k</span><span class="p">]</span><span class="w"> </span><span class="p">[</span><span class="no">:div</span><span class="w"> </span><span class="p">{</span><span class="no">:style</span><span class="w"> </span><span class="p">(</span><span class="nf">gen-style</span><span class="w"> </span><span class="p">(</span><span class="no">:nb-disc</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="p">)</span><span class="w"> </span><span class="n">k</span><span class="p">)}])</span><span class="w"> </span><span class="n">discs</span><span class="p">)])]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-rod</span><span class="w"> </span><span class="s">"left"</span><span class="w"> </span><span class="p">(</span><span class="nf">get-in</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="w"> </span><span class="p">[</span><span class="no">:disc</span><span class="w"> </span><span class="no">:left</span><span class="p">]))</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-rod</span><span class="w"> </span><span class="s">"middle"</span><span class="w"> </span><span class="p">(</span><span class="nf">get-in</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="w"> </span><span class="p">[</span><span class="no">:disc</span><span class="w"> </span><span class="no">:middle</span><span class="p">]))</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-rod</span><span class="w"> </span><span class="s">"right"</span><span class="w"> </span><span class="p">(</span><span class="nf">get-in</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="w"> </span><span class="p">[</span><span class="no">:disc</span><span class="w"> </span><span class="no">:right</span><span class="p">]))]))</span><span class="w">
</span></code></pre>
</div>
<p>Though nothing seem to have changed, now the UI is wired to present the underlying state. Try to update the state manually and see what happen! Enter at REPL (omitting output):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>(swap! c/app-state assoc :disc { :left [2 3 4] :middle [] :right [1] })
(swap! c/app-state assoc :disc { :left [3 4] :middle [2] :right [1] })
(swap! c/app-state assoc :disc { :left [3 4] :middle [1 2] :right [] })
(swap! c/app-state assoc :disc { :left [4] :middle [1 2] :right [3] })
</code></pre>
</div>
<p>(and so on if you fancy)</p>
<p>Notice how the browser’s UI update itself, live, as you enter these commands.</p>
<p>For the next step, we will add control elements to the UI so the user can “play” with it. Add this to the source and save:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">control-panel</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">[</span><span class="no">:span</span><span class="w"> </span><span class="s">"Number of Discs: "</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"num"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"text"</span><span class="p">}]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"reset"</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="s">"Reset"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"button"</span><span class="p">}]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"solve"</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="s">"Solve!"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"button"</span><span class="p">}]])</span><span class="w">
</span></code></pre>
</div>
<p>By the way, notice how our (manually modified) state is preserved even though we have updated our source code.</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Basic_3_State_Persist.png" title="State Persistence" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Basic_3_State_Persist.png" alt="State Persistence" />
</a>
<p class="image-caption">Our App at the end of this subsection. Note that the application state (the number of discs as well as their location) is preserved across live reloading.</p>
</div>
<h3 id="bonus-session">Bonus Session</h3>
<p><em>(What follows is basically more of the same from a setup point of view. You can play along though if you want to also learn the frameworks, since we’ll use some features in a slightly less shallow manner)</em></p>
<p>Now let’s add event handler dynamically. (later we will save it back to the source)</p>
<p>First design a function to construct the <code class="highlighter-rouge">:disc</code> attr (by the way, you may have noticed by now that these long REPL transcripts are somewhat sanitized, but not 100%: I want to highlight how you can always try again and recover from mistakes):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dev:cljs.user=> (range 1 4)
(1 2 3)
dev:cljs.user=> (defn init-tower [n] {:left (range 1 (+ n 1)) :middle [] :right []})
#'cljs.user/init-tower
dev:cljs.user=> (init-tower 5)
{:left (1 2 3 4 5), :middle [], :right []}
dev:cljs.user=> (defn init-tower [n] {:left (apply vector (range 1 (+ n 1))) :middle [] :right []})
#'cljs.user/init-tower
dev:cljs.user=> (init-tower 5)
{:left [1 2 3 4 5], :middle [], :right []}
dev:cljs.user=>
</code></pre>
</div>
<p>Then hook the element (<code class="highlighter-rouge">*1</code> in REPL can be used to refer to the result of last evaluation):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>dev:cljs.user=> (.getElementById js/document "reset")
#object[HTMLInputElement [object HTMLInputElement]]
dev:cljs.user=> (.addEventListener *1 "click" (fn [e] (let [n (.-value (.getElementById js/document "num")) ] (swap! c/app-state assoc :nb-disc n :disc (init-tower n)))) false)
nil
dev:cljs.user=>
</code></pre>
</div>
<p>If you try it out, you’re going to be badly surprised:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Bonus_Fail_1.png" title="Making mistake" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Bonus_Fail_1.png" alt="Making mistake" />
</a>
<p class="image-caption">Epic Fail! Why? (And the rainbow...)</p>
</div>
<p>Why, use the REPL to debug! Examine the state: <code class="highlighter-rouge">@c/app-state</code></p>
<p>It’s strange that the <code class="highlighter-rouge">:disc</code> has such a value. On a moment’s thought, the text field is a string, so we check our suspicion:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>(init-tower "1")
</code></pre>
</div>
<p>Viola, that’s <del>our</del> my mistake. <sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup> Once we know what’s up, fixing it is relatively straight forward, just pass the extracted value through a <code class="highlighter-rouge">js/parseInt</code> call (REPL transcript omitted).</p>
<p>Now that we are done, let’s update the source code:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">control-panel</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">[</span><span class="no">:span</span><span class="w"> </span><span class="s">"Number of Discs: "</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"num"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"text"</span><span class="p">}]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"reset"</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="s">"Reset"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"button"</span><span class="w">
</span><span class="no">:on-click</span><span class="w"> </span><span class="o">#</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="nf">js/parseInt</span><span class="w">
</span><span class="p">(</span><span class="nf">.-value</span><span class="w"> </span><span class="p">(</span><span class="nf">.getElementById</span><span class="w"> </span><span class="n">js/document</span><span class="w"> </span><span class="s">"num"</span><span class="p">)))</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">swap!</span><span class="w"> </span><span class="n">app-state</span><span class="w"> </span><span class="nb">assoc</span><span class="w"> </span><span class="no">:nb-disc</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="no">:disc</span><span class="w"> </span><span class="p">(</span><span class="nf">init-tower</span><span class="w"> </span><span class="n">n</span><span class="p">)))}]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"solve"</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="s">"Solve!"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"button"</span><span class="p">}]])</span><span class="w">
</span></code></pre>
</div>
<p><strong>Exercise:</strong> Wire the button so that it trigger solving the puzzle, playing each step automatically with a reasonable time delay (so mere mortals can see what’s happening). (Hint: read the website for <a href="https://reagent-project.github.io/">reagent</a>)</p>
<h3 id="heads-up-display">Heads Up Display</h3>
<p>I’ve promised to talk about that. So what if we’ve got a mistake in the source code (which we have mostly avoided since we always test things out in the REPL before “committing” them in the source)? Try to edit the handler for the reset button (or add if you didn’t do the bonus session), say by making the handler function’s parentheses unbalanced:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_HUD_Error.png" title="HUD Error example" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_HUD_Error.png" alt="HUD Error example" />
</a>
<p class="image-caption">Sample Error</p>
</div>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_HUD_Warning.png" title="HUD Warning example" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_HUD_Warning.png" alt="HUD Warning example" />
</a>
<p class="image-caption">Sample Warning</p>
</div>
<p>After saving, you will notice a warning window in the browser tab that shows where the error is. Interestingly, the application still works! (Don’t trust me, try to click “Solve!” if you’ve done the exercise in the bonus section. If not, update the state manually in the REPL)</p>
<p>After you’re done, revert the event handler back to the right form.</p>
<h3 id="sourcemap-and-breakpoint">Sourcemap and Breakpoint</h3>
<p>This and the next feature require some simple browser setup. Assuming you’re using Chrome, open the developer console (Ctrl-Shift-I), click the <strong>Customize and control DevTools</strong> button on top right corner (which looks like three dots arranged vertically), and click settings. You will see the following screen:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_EnableFormatter_1.png" title="Chrome DevTools Setting" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_EnableFormatter_1.png" alt="Chrome DevTools Setting" />
</a>
<p class="image-caption">Setting up Chrome for Sourcemap and Custom Formatter</p>
</div>
<p>Make sure that both <strong>“Enable JavaScript source maps”</strong> and <strong>“Enable custom formatters”</strong> are clicked.</p>
<p>Source map is a chrome feature that allows mapping compiled javascript file’s location back to actual source location. The benefit it provides is that when debugging using the browser’s breakpoint, one will now see more meaningful source code.</p>
<p>To test this, edit the function hanoi-solution and add the line:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">hanoi-solution</span><span class="w"> </span><span class="p">[</span><span class="n">n</span><span class="w"> </span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">=</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w">
</span><span class="p">[[</span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">]]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">aux</span><span class="w"> </span><span class="p">(</span><span class="nf">c/find-aux</span><span class="w"> </span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">)</span><span class="w">
</span><span class="n">pre</span><span class="w"> </span><span class="p">(</span><span class="nf">hanoi-solution</span><span class="w"> </span><span class="p">(</span><span class="nb">-</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">src</span><span class="w"> </span><span class="n">aux</span><span class="p">)</span><span class="w">
</span><span class="n">post</span><span class="w"> </span><span class="p">(</span><span class="nf">hanoi-solution</span><span class="w"> </span><span class="p">(</span><span class="nb">-</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="mi">1</span><span class="p">)</span><span class="w"> </span><span class="n">aux</span><span class="w"> </span><span class="n">dst</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nf">js-debugger</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">concat</span><span class="w"> </span><span class="n">pre</span><span class="w"> </span><span class="p">[[</span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">]]</span><span class="w"> </span><span class="n">post</span><span class="p">))))</span><span class="w">
</span></code></pre>
</div>
<p>Now enter the following in REPL:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>(hanoi-solution 4 :left :middle)
</code></pre>
</div>
<p>(Alternatively, edit the event handler for the “Solve!” button to call that function)</p>
<p>Chrome will pause in the debugger:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Sourcemap.png" title="Debugger with sourcemap" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Sourcemap.png" alt="Debugger with sourcemap" />
</a>
<p class="image-caption">An example debugging session in Chrome with sourcemapping. The stacktrace also shows Clojurescript data structures but not the function arguments in the main screen. Read the main exposition for more explanation.</p>
</div>
<p>Notice how clojurescript source, rather than javascript, is shown. Also notice an integer parameter after the filename: this is a feature of our project turned on by the <code class="highlighter-rouge">:source-map-timestamp</code> switch in <code class="highlighter-rouge">project.clj</code>, under the section for <code class="highlighter-rouge">:cljsbuild</code>. Its purpose is to ensure we always get the latest source code, rather than the (HTTP) cached version which may get outdated as we live edit it. (The number is a timestamp so it always change across live/hot reload)</p>
<p>Now look at the call stack, where again clojurescript function is used, as well as the local variables (again located at the right column under the call stack). To see variables in outer scope, you have to click earlier entries in the call stack. Also notice the limitation that custom formatting doesn’t work on the live-inspection of argument passed to function. <sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup></p>
<p>Aside from using clojurescript function, in browser breakpoint also works: click on the line in the find-aux function, and click “continue” to see it pause on that breakpoint as well.</p>
<p>Since this is a recursive function, you may click on “continue” a few more times to see the evolution of evaluation.</p>
<h3 id="custom-formatter-via-cljs-devtools">Custom Formatter via cljs-devtools</h3>
<p>Again, make sure you’ve done the setting from last section. If not, when you type the following to REPL:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="nf">.log</span><span class="w"> </span><span class="n">js/console</span><span class="w"> </span><span class="p">[</span><span class="no">:foo</span><span class="w"> </span><span class="s">"bar"</span><span class="p">])</span><span class="w">
</span></code></pre>
</div>
<p>You’ll see the following screen:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_NoFormatting.png" title="Printing clojurescript data without formatter" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_NoFormatting.png" alt="Printing clojurescript data without formatter" />
</a>
<p class="image-caption">The results of console logging clojurescript data without setting up custom formatting - we instead got the raw, underlying javascript data.</p>
</div>
<p>So if we have done the setup and then enter the following into REPL:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="nf">.log</span><span class="w"> </span><span class="n">js/console</span><span class="w"> </span><span class="n">c/hanoi-tower</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">.log</span><span class="w"> </span><span class="n">js/console</span><span class="w"> </span><span class="err">@</span><span class="n">c/app-state</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">.log</span><span class="w"> </span><span class="n">js/console</span><span class="w"> </span><span class="p">(</span><span class="nf">c/hanoi-solution</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="no">:left</span><span class="w"> </span><span class="no">:middle</span><span class="p">))</span><span class="w">
</span></code></pre>
</div>
<p>We’ll get this:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Formatter.png" title="Printing clojurescript data with formatter" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Formatter.png" alt="Printing clojurescript data with formatter" />
</a>
<p class="image-caption">Custom formatters provided by cljs-devtools, which let us examine Clojurescript data structures and also show other stuffs such as functions nicely.</p>
</div>
<h2 id="devcards">Devcards</h2>
<p>As an alternative to developing your application directly, <a href="https://github.com/bhauman/devcards">Devcards</a> give you an interactive, visual, REPL-like development experience that is like doing demo for each of your application’s component. Perhaps a picture is better here:</p>
<p>Asides from being plain cool, a deeper reason for doing thing this way is to decouple, or separate, the generic components being developed from the application context where it is used. This allows one to explore its state, structure, and behavior more effectively and directly.</p>
<p>To get started, we will scaffold a new project. Go to a new directory and enter:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>lein new devcards devcards-demo
</code></pre>
</div>
<p>(Our project name is <code class="highlighter-rouge">devcards-demo</code>)</p>
<p>The generated project is also a figwheel project, so start it up in the usual way. Then visit the page at <code class="highlighter-rouge">http://localhost:3449/cards.html</code>. You will be greeted with the following screen:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Devcards_Blank1.png" title="Devcards Default 1" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Devcards_Blank1.png" alt="Devcards Default 1" />
</a>
<p class="image-caption">Main page of a Devcards project, with a listing of available pages.</p>
</div>
<p>The project has a default page setup for us. Click on it to see the default card:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Devcards_Blank2.png" title="Devcards Default 2" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Devcards_Blank2.png" alt="Devcards Default 2" />
</a>
<p class="image-caption">A Devcards page. Each card is displayed sequentially.</p>
</div>
<p>Now we want to integrate the code we have so far into it. First update our <code class="highlighter-rouge">project.clj</code> to choose the framework. In our case we chose reagent at the beginning, so update this line:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="no">:dependencies</span><span class="w"> </span><span class="p">[[</span><span class="n">org.clojure/clojure</span><span class="w"> </span><span class="s">"1.8.0"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="n">org.clojure/clojurescript</span><span class="w"> </span><span class="s">"1.9.229"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="n">devcards</span><span class="w"> </span><span class="s">"0.2.3"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="n">sablono</span><span class="w"> </span><span class="s">"0.7.4"</span><span class="p">]</span><span class="w">
</span><span class="c1">;; need to specify this for sablono
</span><span class="w"> </span><span class="c1">;; when not using devcards
</span><span class="w"> </span><span class="p">[</span><span class="n">cljsjs/react</span><span class="w"> </span><span class="s">"15.3.1-0"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="n">cljsjs/react-dom</span><span class="w"> </span><span class="s">"15.3.1-0"</span><span class="p">]</span><span class="w">
</span><span class="o">#</span><span class="n">_</span><span class="p">[</span><span class="n">org.omcljs/om</span><span class="w"> </span><span class="s">"1.0.0-alpha46"</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="n">reagent</span><span class="w"> </span><span class="s">"0.6.0"</span><span class="p">]</span><span class="w"> </span><span class="c1">;; <-- Update this line
</span><span class="w"> </span><span class="p">]</span><span class="w">
</span></code></pre>
</div>
<p>Notice that <code class="highlighter-rouge">#_</code> is the “dispatch” <a href="https://clojure.org/reference/reader#_dispatch">reader macro</a> for ignoring the next form, which can be thought of as a stronger form of commenting. Remember to restart the project after that for lein to download and install new dependencies.</p>
<p>After that, we update the require in the source code:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="nf">ns</span><span class="w"> </span><span class="n">devcards-demo.core</span><span class="w">
</span><span class="p">(</span><span class="no">:require</span><span class="w">
</span><span class="o">#</span><span class="n">_</span><span class="p">[</span><span class="n">om.core</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">om</span><span class="w"> </span><span class="no">:include-macros</span><span class="w"> </span><span class="n">true</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="n">reagent.core</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">reagent</span><span class="p">]</span><span class="w"> </span><span class="c1">;; For the atom
</span><span class="w"> </span><span class="p">[</span><span class="n">clojure.set</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">s</span><span class="p">]</span><span class="w"> </span><span class="c1">;; Our app's require
</span><span class="w"> </span><span class="p">[</span><span class="n">sablono.core</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">sab</span><span class="w"> </span><span class="no">:include-macros</span><span class="w"> </span><span class="n">true</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="no">:require-macros</span><span class="w">
</span><span class="p">[</span><span class="n">devcards.core</span><span class="w"> </span><span class="no">:as</span><span class="w"> </span><span class="n">dc</span><span class="w"> </span><span class="no">:refer</span><span class="w"> </span><span class="p">[</span><span class="n">defcard</span><span class="w"> </span><span class="n">deftest</span><span class="w"> </span><span class="n">defcard-rg</span><span class="p">]]))</span><span class="w"> </span><span class="c1">;; Add defcard-rg
</span></code></pre>
</div>
<p>To bring in the new reagent library and remove om. <sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup></p>
<p>Next, copy the source code of the app we’ve been writing so far into <code class="highlighter-rouge">core.cljs</code>, in between <code class="highlighter-rouge">(enable-console-print!)</code> and the <code class="highlighter-rouge">defcard</code>, while taking care to <em>still</em> use Reagent’s atom implementation (as we’re not <code class="highlighter-rouge">refer</code>ing the <code class="highlighter-rouge">atom</code> in reagent namespace anymore), so our state should instead look like:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="nf">defonce</span><span class="w"> </span><span class="n">app-state</span><span class="w"> </span><span class="p">(</span><span class="nf">reagent/atom</span><span class="w"> </span><span class="p">{</span><span class="no">:text</span><span class="w"> </span><span class="s">"Hello world!"</span><span class="w"> </span><span class="no">:nb-disc</span><span class="w"> </span><span class="mi">4</span><span class="w"> </span><span class="no">:disc</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="no">:left</span><span class="w"> </span><span class="p">[</span><span class="mi">1</span><span class="w"> </span><span class="mi">2</span><span class="w"> </span><span class="mi">3</span><span class="w"> </span><span class="mi">4</span><span class="p">]</span><span class="w"> </span><span class="no">:middle</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="no">:right</span><span class="w"> </span><span class="p">[]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">}))</span><span class="w">
</span></code></pre>
</div>
<p>Then we want to create a new card that holds our whole app, and to do so require using the interface correctly (with reagent integration).</p>
<p>Fortunately the library has a <a href="http://rigsomelight.com/devcards/#!/devdemos.reagent">guide</a> on how to do that. The gist of it is that since Devcards is designed to handle <strong>ReactElement</strong> for DOM-tree (alongside support for displaying plain Clojurescript data structure), we need to call <code class="highlighter-rouge">reagent.core/as-element</code> to convert reagent code into those elements. The macro <code class="highlighter-rouge">devcards.core/defcard-rg</code> do this and in a smart way that handles the interface details (like when we pass a function that returns DOM-tree like object instead) automagically.</p>
<p>By the way the guide above also has documentations for various aspects of Devcards - you may browse around through the navigation menu at the top of the page.</p>
<p>Add the following code:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">hanoi-tower-widget</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">[</span><span class="no">:h1</span><span class="w"> </span><span class="p">(</span><span class="no">:text</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="p">)]</span><span class="w">
</span><span class="p">(</span><span class="nf">hanoi-tower</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">control-panel</span><span class="p">)</span><span class="w">
</span><span class="p">])</span><span class="w">
</span><span class="p">(</span><span class="nf">defcard-rg</span><span class="w"> </span><span class="n">hanoi-tower-card</span><span class="w">
</span><span class="p">[</span><span class="n">hanoi-tower-widget</span><span class="p">]</span><span class="w">
</span><span class="n">app-state</span><span class="w">
</span><span class="p">{</span><span class="no">:inspect-data</span><span class="w"> </span><span class="n">true</span><span class="w">
</span><span class="no">:frame</span><span class="w"> </span><span class="n">true</span><span class="w">
</span><span class="no">:history</span><span class="w"> </span><span class="n">true</span><span class="p">})</span><span class="w">
</span></code></pre>
</div>
<p>Now we have our app embedded in as a card:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Devcards_Integration.png" title="Integrating our app into Devcards" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Devcards_Integration.png" alt="Integrating our app into Devcards" />
</a>
<p class="image-caption">Adding our app as a card. Note that the current state is displayed below.</p>
</div>
<p>The last argument in our <code class="highlighter-rouge">defcard-rg</code> macro call controls the meta-feature provided by Devcards. <code class="highlighter-rouge">inspect-data</code> give us a view into the current component state as the screen capture above shows; <code class="highlighter-rouge">frame</code> simply add a border around the card; <code class="highlighter-rouge">history</code> records the history of state change and allows one to “time-travel” (One of the super-power of the functional-reactive style, enabled by completely teasing out the state and controlling its mutation strictly) - we will see this play out later.</p>
<p>Suppose we want to enhance this widget so that user can play with the puzzle themselves by supplying their own moves. First we add the UI: add the following code for generating a disc selection menu:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">disc-select</span><span class="w"> </span><span class="p">[</span><span class="n">id-prefix</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">letfn</span><span class="w"> </span><span class="p">[(</span><span class="nf">disc-item</span><span class="w"> </span><span class="p">[</span><span class="n">id</span><span class="w"> </span><span class="n">input-name</span><span class="w"> </span><span class="n">label</span><span class="w"> </span><span class="nb">val</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="no">:li</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="n">id</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"radio"</span><span class="w"> </span><span class="no">:name</span><span class="w"> </span><span class="n">input-name</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="nb">val</span><span class="p">}]</span><span class="w">
</span><span class="p">[</span><span class="no">:label</span><span class="w"> </span><span class="p">{</span><span class="no">:for</span><span class="w"> </span><span class="n">id</span><span class="p">}</span><span class="w"> </span><span class="n">label</span><span class="p">]])]</span><span class="w">
</span><span class="p">[</span><span class="no">:ul</span><span class="w"> </span><span class="p">{</span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:display</span><span class="w"> </span><span class="s">"inline-block"</span><span class="w"> </span><span class="no">:list-style</span><span class="w"> </span><span class="s">"none"</span><span class="w"> </span><span class="no">:padding-right</span><span class="w"> </span><span class="s">"20px"</span><span class="p">}}</span><span class="w">
</span><span class="p">(</span><span class="nf">disc-item</span><span class="w"> </span><span class="p">(</span><span class="nb">str</span><span class="w"> </span><span class="n">id-prefix</span><span class="w"> </span><span class="s">"-left"</span><span class="p">)</span><span class="w"> </span><span class="n">id-prefix</span><span class="w"> </span><span class="s">"Left Disc"</span><span class="w"> </span><span class="s">"left"</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">disc-item</span><span class="w"> </span><span class="p">(</span><span class="nb">str</span><span class="w"> </span><span class="n">id-prefix</span><span class="w"> </span><span class="s">"-middle"</span><span class="p">)</span><span class="w"> </span><span class="n">id-prefix</span><span class="w"> </span><span class="s">"Middle Disc"</span><span class="w"> </span><span class="s">"middle"</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nf">disc-item</span><span class="w"> </span><span class="p">(</span><span class="nb">str</span><span class="w"> </span><span class="n">id-prefix</span><span class="w"> </span><span class="s">"-right"</span><span class="p">)</span><span class="w"> </span><span class="n">id-prefix</span><span class="w"> </span><span class="s">"Right Disc"</span><span class="w"> </span><span class="s">"right"</span><span class="p">)]</span><span class="w">
</span><span class="p">))</span><span class="w">
</span></code></pre>
</div>
<p>Then update the control panel (with a dummy event handler):</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">hanoi-ui-move!</span><span class="w"> </span><span class="p">[</span><span class="n">e</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">do</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">control-panel</span><span class="w"> </span><span class="p">[]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w">
</span><span class="p">[</span><span class="no">:span</span><span class="w"> </span><span class="s">"Number of Discs: "</span><span class="p">]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"num"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"text"</span><span class="p">}]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"reset"</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="s">"Reset"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"button"</span><span class="w">
</span><span class="no">:on-click</span><span class="w"> </span><span class="o">#</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">n</span><span class="w"> </span><span class="p">(</span><span class="nf">js/parseInt</span><span class="w">
</span><span class="p">(</span><span class="nf">.-value</span><span class="w"> </span><span class="p">(</span><span class="nf">.getElementById</span><span class="w"> </span><span class="n">js/document</span><span class="w"> </span><span class="s">"num"</span><span class="p">)))</span><span class="w"> </span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">swap!</span><span class="w"> </span><span class="n">app-state</span><span class="w"> </span><span class="nb">assoc</span><span class="w"> </span><span class="no">:nb-disc</span><span class="w"> </span><span class="n">n</span><span class="w"> </span><span class="no">:disc</span><span class="w"> </span><span class="p">(</span><span class="nf">init-tower</span><span class="w"> </span><span class="n">n</span><span class="p">)))}]</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"solve"</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="s">"Solve!"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"button"</span><span class="p">}]]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w"> </span><span class="p">{</span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:display</span><span class="w"> </span><span class="s">"table"</span><span class="p">}}</span><span class="w">
</span><span class="p">[</span><span class="no">:span</span><span class="w"> </span><span class="p">{</span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:display</span><span class="w"> </span><span class="s">"table-cell"</span><span class="w"> </span><span class="no">:vertical-align</span><span class="w"> </span><span class="s">"middle"</span><span class="p">}}</span><span class="w"> </span><span class="s">"From:"</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">disc-select</span><span class="w"> </span><span class="s">"from"</span><span class="p">)</span><span class="w">
</span><span class="p">[</span><span class="no">:span</span><span class="w"> </span><span class="p">{</span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:display</span><span class="w"> </span><span class="s">"table-cell"</span><span class="w"> </span><span class="no">:vertical-align</span><span class="w"> </span><span class="s">"middle"</span><span class="p">}}</span><span class="w"> </span><span class="s">"To:"</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">disc-select</span><span class="w"> </span><span class="s">"to"</span><span class="p">)</span><span class="w">
</span><span class="p">[</span><span class="no">:span</span><span class="w"> </span><span class="p">{</span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:display</span><span class="w"> </span><span class="s">"table-cell"</span><span class="w"> </span><span class="no">:vertical-align</span><span class="w"> </span><span class="s">"middle"</span><span class="p">}}</span><span class="w">
</span><span class="p">[</span><span class="no">:input</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"move"</span><span class="w"> </span><span class="no">:value</span><span class="w"> </span><span class="s">"Move!"</span><span class="w"> </span><span class="no">:type</span><span class="w"> </span><span class="s">"button"</span><span class="w">
</span><span class="no">:on-click</span><span class="w"> </span><span class="n">hanoi-ui-move!</span><span class="p">}]]]</span><span class="w">
</span><span class="p">[</span><span class="no">:div</span><span class="w"> </span><span class="p">{</span><span class="no">:id</span><span class="w"> </span><span class="s">"hanoi-error"</span><span class="w"> </span><span class="no">:style</span><span class="w"> </span><span class="p">{</span><span class="no">:color</span><span class="w"> </span><span class="s">"red"</span><span class="w"> </span><span class="no">:background-color</span><span class="w"> </span><span class="s">"#f1f1f1"</span><span class="p">}}]])</span><span class="w">
</span></code></pre>
</div>
<p>After that, develop the pure functions for validating the move and performing the move (which is a function from current state to next state). This should be done via the REPL as before, for reference below is one solution:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">move-disc</span><span class="w"> </span><span class="p">[</span><span class="n">discs</span><span class="w"> </span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">item</span><span class="w"> </span><span class="p">(</span><span class="nb">first</span><span class="w"> </span><span class="p">(</span><span class="nf">src</span><span class="w"> </span><span class="n">discs</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="nb">-></span><span class="w"> </span><span class="n">discs</span><span class="w">
</span><span class="p">(</span><span class="nf">update</span><span class="w"> </span><span class="n">src</span><span class="w"> </span><span class="o">#</span><span class="p">(</span><span class="nb">apply</span><span class="w"> </span><span class="nb">vector</span><span class="w"> </span><span class="p">(</span><span class="nb">rest</span><span class="w"> </span><span class="n">%</span><span class="p">)))</span><span class="w">
</span><span class="p">(</span><span class="nf">update</span><span class="w"> </span><span class="n">dst</span><span class="w"> </span><span class="o">#</span><span class="p">(</span><span class="nb">apply</span><span class="w"> </span><span class="nb">vector</span><span class="w"> </span><span class="p">(</span><span class="nb">cons</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="n">%</span><span class="p">))))))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">validate-move</span><span class="w"> </span><span class="p">[</span><span class="n">discs</span><span class="w"> </span><span class="n">src</span><span class="w"> </span><span class="n">dst</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nb">if-let</span><span class="w"> </span><span class="p">[</span><span class="n">item</span><span class="w"> </span><span class="p">(</span><span class="nb">first</span><span class="w"> </span><span class="p">(</span><span class="nf">src</span><span class="w"> </span><span class="n">discs</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="nb">or</span><span class="w"> </span><span class="p">(</span><span class="nf">empty?</span><span class="w"> </span><span class="p">(</span><span class="nf">dst</span><span class="w"> </span><span class="n">discs</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="nb"><</span><span class="w"> </span><span class="n">item</span><span class="w"> </span><span class="p">(</span><span class="nb">first</span><span class="w"> </span><span class="p">(</span><span class="nf">dst</span><span class="w"> </span><span class="n">discs</span><span class="p">))))</span><span class="w">
</span><span class="n">false</span><span class="p">))</span><span class="w">
</span></code></pre>
</div>
<p>Again testing it out in REPL:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>devcards:cljs.user=> (move-disc {:left '(1 2 3 4) :middle '() :right '()} :left :right)
{:left [2 3 4], :middle (), :right [1]}
devcards:cljs.user=> (move-disc {:left '(3) :middle '(2 4) :right '(1)} :middle :left)
{:left [2 3], :middle [4], :right (1)}
devcards:cljs.user=> (validate-move {:left '(2) :middle '(1 4) :right '(3)} :left :right)
true
devcards:cljs.user=> (validate-move {:left '(2) :middle '(1 3 4) :right '()} :middle :right)
true
devcards:cljs.user=> (validate-move {:left '(2) :middle '(1 3 4) :right '()} :right :left)
false
devcards:cljs.user=> (validate-move {:left '() :middle '(3 4) :right '(1 2)} :middle :right)
false
</code></pre>
</div>
<p>Finally, we tie in everything together and tidy up some bits with the following code for acting on the UI as well as the event handler itself:</p>
<div class="language-clojure highlighter-rouge"><pre class="highlight"><code><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">rod-str->label</span><span class="w"> </span><span class="p">[</span><span class="n">str-val</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">case</span><span class="w"> </span><span class="n">str-val</span><span class="w">
</span><span class="s">"left"</span><span class="w"> </span><span class="no">:left</span><span class="w">
</span><span class="s">"middle"</span><span class="w"> </span><span class="no">:middle</span><span class="w">
</span><span class="s">"right"</span><span class="w"> </span><span class="no">:right</span><span class="w">
</span><span class="n">nil</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">get-radio-val</span><span class="w"> </span><span class="p">[</span><span class="n">radio-name</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">el</span><span class="w"> </span><span class="p">(</span><span class="nf">.querySelector</span><span class="w"> </span><span class="n">js/document</span><span class="w">
</span><span class="p">(</span><span class="nb">str</span><span class="w"> </span><span class="s">"input[name=\""</span><span class="w"> </span><span class="n">radio-name</span><span class="w"> </span><span class="s">"\"]:checked"</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="k">if</span><span class="w"> </span><span class="p">(</span><span class="nb">nil?</span><span class="w"> </span><span class="n">el</span><span class="p">)</span><span class="w">
</span><span class="n">nil</span><span class="w">
</span><span class="p">(</span><span class="nf">.-value</span><span class="w"> </span><span class="n">el</span><span class="p">))))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">set-err-msg</span><span class="w"> </span><span class="p">[</span><span class="n">msg</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="nf">set!</span><span class="w"> </span><span class="p">(</span><span class="nf">.-innerHTML</span><span class="w"> </span><span class="p">(</span><span class="nf">.getElementById</span><span class="w"> </span><span class="n">js/document</span><span class="w"> </span><span class="s">"hanoi-error"</span><span class="p">))</span><span class="w"> </span><span class="n">msg</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="k">defn</span><span class="w"> </span><span class="n">hanoi-ui-move!</span><span class="w"> </span><span class="p">[</span><span class="n">e</span><span class="p">]</span><span class="w">
</span><span class="p">(</span><span class="k">let</span><span class="w"> </span><span class="p">[</span><span class="n">from-val</span><span class="w"> </span><span class="p">(</span><span class="nf">rod-str->label</span><span class="w"> </span><span class="p">(</span><span class="nf">get-radio-val</span><span class="w"> </span><span class="s">"from"</span><span class="p">))</span><span class="w">
</span><span class="n">to-val</span><span class="w"> </span><span class="p">(</span><span class="nf">rod-str->label</span><span class="w"> </span><span class="p">(</span><span class="nf">get-radio-val</span><span class="w"> </span><span class="s">"to"</span><span class="p">))]</span><span class="w">
</span><span class="p">(</span><span class="k">cond</span><span class="w">
</span><span class="p">(</span><span class="nb">nil?</span><span class="w"> </span><span class="n">from-val</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">set-err-msg</span><span class="w"> </span><span class="s">"Please select from disc."</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">nil?</span><span class="w"> </span><span class="n">to-val</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">set-err-msg</span><span class="w"> </span><span class="s">"Please select to disc."</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">=</span><span class="w"> </span><span class="n">from-val</span><span class="w"> </span><span class="n">to-val</span><span class="p">)</span><span class="w"> </span><span class="p">(</span><span class="nf">set-err-msg</span><span class="w"> </span><span class="s">"From disc should be different from to disc."</span><span class="p">)</span><span class="w">
</span><span class="p">(</span><span class="nb">not</span><span class="w"> </span><span class="p">(</span><span class="nf">validate-move</span><span class="w"> </span><span class="p">(</span><span class="no">:disc</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="p">)</span><span class="w"> </span><span class="n">from-val</span><span class="w"> </span><span class="n">to-val</span><span class="p">))</span><span class="w"> </span><span class="p">(</span><span class="nf">set-err-msg</span><span class="w"> </span><span class="s">"Invalid move."</span><span class="p">)</span><span class="w">
</span><span class="no">:else</span><span class="w"> </span><span class="p">(</span><span class="nf">do</span><span class="w">
</span><span class="p">(</span><span class="nf">swap!</span><span class="w"> </span><span class="n">app-state</span><span class="w"> </span><span class="nb">assoc</span><span class="w"> </span><span class="no">:disc</span><span class="w"> </span><span class="p">(</span><span class="nf">move-disc</span><span class="w"> </span><span class="p">(</span><span class="no">:disc</span><span class="w"> </span><span class="err">@</span><span class="n">app-state</span><span class="p">)</span><span class="w"> </span><span class="n">from-val</span><span class="w"> </span><span class="n">to-val</span><span class="p">))</span><span class="w">
</span><span class="p">(</span><span class="nf">set-err-msg</span><span class="w"> </span><span class="s">""</span><span class="p">)))))</span><span class="w">
</span></code></pre>
</div>
<p>Below is our finished app:</p>
<div class="image-wrapper">
<a href="/assets/posts/getting-started-with-clojurescript-development-setup/CLJS_Devcards_History.png" title="Finished app with Devcards History feature" target="_blank">
<img src="/assets/posts/getting-started-with-clojurescript-development-setup/thumbnails/CLJS_Devcards_History.png" alt="Finished app with Devcards History feature" />
</a>
<p class="image-caption">Final state of our Devcards app. It supports user moves. Notice the highlighted tooltip for time-travelling.</p>
</div>
<h2 id="endnotes">Endnotes</h2>
<p>Phew! This is a pretty long post for an introductory topic! I hope you enjoyed it. It turns out that there are other articles/series on the web that offer something similar, and I heartily recommend the following if you’re interested for more:</p>
<ul>
<li>This <a href="https://practicalli.github.io/clojurescript/">workshop</a> is more comprehensive/deep in its coverage of topics and worth working through if you want to deep dive, head first, into actually using Clojurescript.</li>
<li><a href="http://www.alexeberts.com/exploring-the-clojurescript-repl/">Exploring the Clojurescript REPL</a> is somewhat similar in scope to this article, the difference being that it gives more in-depth explanation of both language level as well as library level mechanism.</li>
</ul>
<p>Until then! See you!</p>
<p><em>(To be continued…)</em></p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:1">
<p>See <a href="https://hackernoon.com/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f">How it feels to learn JavaScript in 2016</a>, although even the Javascript community has recognized this problem and is working on it, hence the response <a href="https://medium.com/front-end-hacking/how-it-feels-to-learn-javascript-in-2017-a934b801fbe">How it feels to learn JavaScript in 2017</a>. (But it doesn’t cover build tools…) <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Javascript is ubiquitous, but its original version is done over the span of a bit more than a <a href="https://en.wikipedia.org/wiki/Brendan_Eich">week</a>, and so there are many <a href="https://www.youtube.com/watch?v=2pL28CcEijU">rough</a> edges. For example, its type system is… <a href="https://www.reddit.com/r/ProgrammerHumor/comments/5bemtx/christianity_and_javascript_xpost/">subtle</a>. (In defense though it is good practise to use <a href="https://stackoverflow.com/questions/5101948/javascript-checking-for-null-vs-undefined-and-difference-between-and">strict</a> equality to avoid these headaches) If this is too religious (in <em>both</em> meanings in this context) for your taste, a simpler pain point is the lack of module/namespace/package in the original language, with the results being <a href="https://stackoverflow.com/questions/16521471/relation-between-commonjs-amd-and-requirejs">this</a>. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>Given how other ecosystem have been catching up, in theory these difference aren’t intrinsic anymore. However, it could still be argued that these desirable traits are more easily/naturally achieved in Clojurescript than other languages. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>Don’t laugh, this is the actual mistake I made while preparing this tutorial. In fact I struggled a lot on this part due to all the fuse about interoping with Javascript directly, without the help of even a jQuery analog (There are libraries available, but you will need to restart the whole application due to <code class="highlighter-rouge">project.clj</code> change). <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
<li id="fn:5">
<p>This is quite a pity - to resolve this we may use the <a href="https://github.com/binaryage/dirac">Dirac</a> Chrome extension which I plan to write in the next post in this series. <a href="#fnref:5" class="reversefootnote">↩</a></p>
</li>
<li id="fn:6">
<p>For some reason even restarting doesn’t work for me, so I entered <code class="highlighter-rouge">(reload-config)</code> at figwheel’s REPL and then force refresh the browser’s page (Ctrl-F5) to truly rebuild everything. <a href="#fnref:6" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>We have seen miraculous progress in frontend development in these few years - the domination/monopoly of javascript, AJAX, node.js, the rise and rapid iteration of javascript libraries/frameworks and Single Page Application (with the current crop being Angular, React, Vue, as well as many others). But we’ve also experienced growing pain - the churn rate of libraries is way too high1, javascript as a language (and as part of the toolchain) is too hopelessly ill-equipped2 to deal with the demand of modern frontend space, with attendant attempts to either patch over (transpiling languages such as Coffeescript and Typescript), reform (ES-whatever), or outright replace (WebAssembly) it. Clojurescript is a solution in the “patch-over” category that bring the elegance and pleasure of writing program in Clojure to the web. More importantly, Clojurescript has placed a strong emphasis on Interactive development, and has been leading other technology stack over the years. See How it feels to learn JavaScript in 2016, although even the Javascript community has recognized this problem and is working on it, hence the response How it feels to learn JavaScript in 2017. (But it doesn’t cover build tools…) ↩ Javascript is ubiquitous, but its original version is done over the span of a bit more than a week, and so there are many rough edges. For example, its type system is… subtle. (In defense though it is good practise to use strict equality to avoid these headaches) If this is too religious (in both meanings in this context) for your taste, a simpler pain point is the lack of module/namespace/package in the original language, with the results being this. ↩The Monad Tutorial Fallacy, Part Two: Monad, Kleisli and Eilenberg-Moore Category2017-10-03T20:36:00+08:002017-10-03T20:36:00+08:00https://lemonteaa.github.io/fundamentals/2017/10/03/the-monad-tutorial-fallacy-part-two-monad-kleisli-and-eilenberg-moore-category<p><em>(Go to <a href="/fundamentals/2017/08/28/the-monad-tutorial-fallacy-part-one-introduction-prequel.html">Part 1</a> of this series)</em></p>
<p>In part 2 of this series we explain the technical core of the theory of Monad. Of central importance is the attempt to provide algebraic interpretation of everything, as well as to convert things into suitably algebraic structures. We will also revisit this issue at an arguably “better” (but more advanced) angle in part 5.</p>
<p>While the picture for an adjunction is relatively clear, it suffers from needing to work with two categories at the same time. Monad fixes this by composing the left and right adjoint functors into an endofunctor <script type="math/tex">T = G \circ F</script>, <script type="math/tex">T: \mathcal{C} \rightarrow \mathcal{C}</script> over a single category. An additional advantage with this construction is that it is more algebraic as it is now composable: we can iterate <script type="math/tex">T</script> to get <script type="math/tex">T^2</script>, <script type="math/tex">T^3</script> etc for example. <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>
<!--more--></p>
<h2 id="on-the-benefit-of-being-algebraicfunctional">On the benefit of being Algebraic/Functional</h2>
<p>It is perhaps appropriate at this point to pontificate on what does it means to be “algebraic”, and why do we obsesses over it, before proceeding further on the details. From my armchair philosophical perspective, being “algebraic” simply mean having a system of rules for manipulating symbols. But this “definition” is obviously wrong, for with this requirement the whole of pure mathematics at the very least qualifies! (All math are built from logic, and formal logic is just such a system of rules, albeit a bit more complicated than arithmetic) What distinguishes algebra from other such formal system is an intrinsic quality that it is easy and convenient to use.</p>
<p>But “easy” is subjective. And there seems to be more than one kind of feature that contribute to this elusive quality. These features include:</p>
<ul>
<li>The rules and system should be as Context-free as possible, so that operations can be liberally applied, and so that a piece of operation/reasoning carried out somewhere can be taken and re-applied anywhere else
<ul>
<li>i.e. Equational Reasoning</li>
<li>And in other words things should be reusable.</li>
</ul>
</li>
<li>Composability. (Excuse my circular reasoning) It should be possible to chain functions together and if these functions are expressed with certain specific structures in mind, these structures should be preserved after composition.
<ul>
<li>Moreover, it should ideally have the Compositionality property - the meaning of a composed function should be derivable from the meaning of the individual functions alone.</li>
<li>What they imply are that you have an easy way to build larger structures out of smaller one, while controlling (semantic) complexity.</li>
</ul>
</li>
</ul>
<p>If you are a programmer, you may find this list suspiciously familiar even though you may have forgotten all the math since you’re done with high school. Basically they are just the benefits of functional programming: pure functions are composable, safe, reusable, and easy to reason about and understand.</p>
<h2 id="defining-monad-is-a-dangerous-affair">Defining Monad is a Dangerous Affair</h2>
<p>If one dig around a bit in Haskell’s community, one can easily see how this is a topic laden with traps, controversies, and a sea of confused/perplexed beginners. There is an active advise against posting new Monad tutorials. There is a trail of <a href="https://wiki.haskell.org/Monad_tutorials_timeline">past attempts</a> at such tutorials. There is a post in the wiki specifically to debunk what <a href="https://wiki.haskell.org/What_a_Monad_is_not">Monad is not</a> and notwithstanding such effort, it is (in this author’s opinion) next to impossible to eradicate those “wrong” definitions, not even in principle. Oh and if you try to read Monad tutorial (like this one), you will find out that everyone’s definition is <a href="http://www.codecommit.com/blog/ruby/monads-are-not-metaphors">different</a>. <sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup></p>
<p>That’s quite a downer, to put it mildly. Why? <sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup></p>
<p><em><begin rant></em></p>
<p>Please bear with me as I do a second round of philosophizing and go meta. Defining real world, nebulous concepts is almost always messy (in contrast to the pristine one you get in pure math). They are typically complicated. Have multiple facets. Have multiple possible perspectives to look at that are mostly equally reasonable within their own frameworks. (Or, at least, no one managed to convince anyone else in case of a conflict) Have multiple layers of understanding possible to attain. Ad Nauseum.</p>
<p>With this kind of stuff, saying any particular definition is “wrong” is meaningless without defining “wrongness” within a suitable context. Say in a hypothetical (i.e. oversimplified) situation there is a hierarchy of definitions possible, from the most specific to the most general. Is the specific one wrong because it fails to capture the full range of possibilities, or the general one wrong because it is utterly useless for application? Sometimes I think we geeks and nerds need to stop pretending math/logic/programming is the solo, absolute standard of truth in all domains. Just go read some (ok, any) sociology texts and resists the temptation to call everything in it bullshit. (except some toxic part of postmodernism maybe?)</p>
<p><em><end rant></em></p>
<p>But it seems irresponsible to go on without any definition at all besides the formal one. So, in light of the sensitiveness of this issue, I will tread carefully and give more than one definitions.</p>
<p>A Monad is not about statefulness/impurity. It’s also not about explicit sequencing of execution. They are, however, possible applications of Monad. (Among others)</p>
<h3 id="most-general-definition-i-can-think-of">Most general definition I can think of</h3>
<p>(Inspired by this <a href="https://www.reddit.com/r/haskell/comments/68fb19/monad_tutorial_no_57005/dgynjik/">brilliant comment</a> over there)</p>
<blockquote>
<p>Monad is a mathematical theory that can be used to extend the semantics of a programming language in a principled manner, provided the extra structures admits an algebraic formulation.</p>
</blockquote>
<p>Because of this I believe the closest comparison one can make is versus Macro in the Lisp family of language <sup id="fnref:4"><a href="#fn:4" class="footnote">4</a></sup>. At least they are roughly at the same conceptual level of abstraction.</p>
<p>One should also be careful to distinguish between Monad-as-a-concept, versus how it is implemented in actual languages.</p>
<p>Monad is not a <a href="http://lambda-diode.com/programming/monads-are-a-class-of-hard-drugs">silver bullet</a> nor black magic. It is, in the final analysis, up to you to decide what to do with it.</p>
<h3 id="operational-definition---full-buy-in">Operational definition - Full buy in</h3>
<blockquote>
<p>A Monad is a contract between the programmer and the language. Programmer agrees to structure their program in a particular form, and to supply proof of the Monad axiom themselves, outside of the program. In return, the language promises automatic guarantee of some properties of the program.</p>
</blockquote>
<p>But asking a programmer to write proof is scary, horrible stuff. Monad is thoughtful in this regard:
As there is multiple equivalent formulation of Monad, one have the option to choose which one to work on.
The math statements that one is actually required to prove is also algebraic/equational in nature, so it is in principle possible to just compute/derive with lambda calculus blindly and hope that the equations just line up, as opposed to requiring professional grade mathematical literacy.</p>
<h3 id="operational-definition---partial-buy-in">Operational definition - Partial buy in</h3>
<p>In case even that is too much, it is also possible to write code in a monadic style. In this case one adopts patterns inspired by the Monad, but refrain from (actively) doing the proof or making their code completely compliant to the requirement. One may still gain some benefits, but foregoes the mathematical guarantee offered by the theory.</p>
<h2 id="formal-definition-of-monad">Formal definition of Monad</h2>
<p>For delicate, highly abstract entity like Monad (as well as other heavily categorical construct, and especially things involving <a href="https://ncatlab.org/nlab/show/higher+category+theory">higher category</a>), one can argue that the only safe way to grok it would be to look at the full, formal definition (otherwise one runs the risk of missing a “small” piece of the definition - which usually contains a “large” number of such pieces - that turns out to be crucial in unexpected place). But doing so come with the cost of things being opaque and complicated. While I may not be able to solve the “complicated” part (irreducible complexity, say), I have tried to deal with the first by presenting a mixture of motivation, discourse, supporting theories, etc.</p>
<p>Even so, we won’t try to understand Monad all in one go, preferring an iterative and indirect approach instead. We will provide a preliminary, algebraic understanding of Monad, only in the context of classical Mathematics (through the “Universal” example of Free algebra vs Forgetful Functor). In part 3 of this series we will give an interpretation within the context of raw vs structured computational model, alongside actual examples of Monad in programming, while in part 4 we will give alternative, equivalent formulation of Monad that admits an easier interpretation in the computational context. (Part 5’s purpose has been stated at the beginning of this article) Nonetheless, all these niceties depends on the core theory presented in this part.</p>
<p>The actual definition of <a href="https://en.wikipedia.org/wiki/Monad_(category_theory)">Monad</a> is here (Don’t worry if you don’t get it all at once! The important thing in this part is actually the theory of Kleisli and Eilenberg-Moore Category since I don’t see them fully interpreted anywhere except for some handwaving. Just read on and stay tuned for part 3 - 5 :P ):</p>
<ul>
<li>A Monad over a category <script type="math/tex">\mathcal{C}</script> is a triple <script type="math/tex">(T, \eta, \mu)</script> where <script type="math/tex">T</script> is a functor <script type="math/tex">\mathcal{C} \rightarrow \mathcal{C}</script>.
<ul>
<li>(for those not studying math: I must highlight that a functor when implemented in a programming language is actually two things: <script type="math/tex">T</script> here maps objects in <script type="math/tex">\mathcal{C}</script> to objects in <script type="math/tex">\mathcal{C}</script>, and <strong>also</strong> morphisms/function in <script type="math/tex">\mathcal{C}</script> to morphisms/function in <script type="math/tex">\mathcal{C}</script>, where the (co)domain object is also mapped. That is, <script type="math/tex">T</script> sends <script type="math/tex">f: X \rightarrow Y</script> into <script type="math/tex">Tf: TX \rightarrow TY</script>)</li>
</ul>
</li>
<li><script type="math/tex">\eta</script> is the unit (sometimes called <code class="highlighter-rouge">return</code> in programming), which is a natural transformation from the identity functor to <script type="math/tex">\mathcal{C}</script>. More concretely for any object <script type="math/tex">X</script> in <script type="math/tex">\mathcal{C}</script>, we have a morphism/function <script type="math/tex">\eta_X: X \rightarrow TX</script>.</li>
<li>And <script type="math/tex">\mu</script> is the “multiplication” / “algebraic operation” (called <code class="highlighter-rouge">join</code> or <code class="highlighter-rouge">flatten</code> in programming). It is also a natural transformation, this time from <script type="math/tex">T^2</script> to <script type="math/tex">T</script>. That is it is a polymorphic function from <script type="math/tex">T^2 X</script> to <script type="math/tex">TX</script>.
<ul>
<li>Notice that both <a href="https://en.wikipedia.org/wiki/Functor#Definition">functor</a> and <a href="https://en.wikipedia.org/wiki/Natural_transformation#Definition">natural transformation</a> obey certain constraints/equations/axioms, which are not listed here.</li>
</ul>
</li>
<li>Finally, our <script type="math/tex">\eta</script> and <script type="math/tex">\mu</script> satisfies two additional coherence conditions, expressed via commutative diagrams:</li>
</ul>
<script type="math/tex; mode=display">% <![CDATA[
\array{
\phantom{\mu T} T^3 & \stackrel{T \mu}\longrightarrow & T^2 \\
^{\mu T}\downarrow & & \downarrow^\mu \\
\phantom{\mu T} T^2 & \stackrel{\mu}\longrightarrow & T
}
\qquad
\array{
\phantom{T \eta} T & \stackrel{\eta T}\longrightarrow & T^2 \\
^{T \eta} \downarrow & \searrow^\text{id} & \downarrow^\mu \\
\phantom{T \eta} T^2 & \stackrel{\mu}\longrightarrow & T
} %]]></script>
<p>As many people/articles/tutorials have observed, these laws look a lots like the identity and associativity law of an algebra (say a group):</p>
<ul>
<li>(Identity) For all <script type="math/tex">x</script>, <script type="math/tex">0 + x = x + 0 = x</script>.</li>
<li>(Associativity) For all <script type="math/tex">x, y, z</script>, <script type="math/tex">x + (y + z) = (x + y) + z</script>.</li>
</ul>
<p>We will in time comes to a fuller understanding of this point, through multiple rounds of seeing this play out in different level of abstractions.</p>
<p>Before moving on, here’s a technical aside (included since we’ll assume knowledge of this bit when doing proofs later on in this post): the notation mixing functor and natural transform above is a shorthand. Suppose <script type="math/tex">\epsilon: F \Rightarrow G</script> is a natural transform from <script type="math/tex">F</script> to <script type="math/tex">G</script>, and let <script type="math/tex">H</script> be another functor. Then <script type="math/tex">H\epsilon</script> means applying the functor <script type="math/tex">H</script> to the arrow <script type="math/tex">FX \stackrel{\epsilon_X}\longrightarrow GX</script> to get <script type="math/tex">H(FX) \stackrel{H \epsilon_X}\longrightarrow H(GX)</script>. Meanwhile, <script type="math/tex">\epsilon H</script> actually means the natural transform applied at the object <script type="math/tex">HX</script> instead of <script type="math/tex">X</script>, i.e. <script type="math/tex">F(HX) \stackrel{\epsilon_{HX}}\longrightarrow G(HX)</script>.</p>
<p>We still need to connect this formal definition to our earlier motivation of composing adjoint functors. In fact there is a general construction to get a monad from any adjoint functors. <script type="math/tex">T</script> is just <script type="math/tex">G \circ F</script> as mentioned before. The unit is just the unit in the adjunction. The join operation is obtained by applying the counit to <script type="math/tex">T^2</script>, that is: <script type="math/tex">T^2 = (GF)(GF) = G(FG)F \longrightarrow G \operatorname{1}_{\mathcal{D}} F = GF = T</script>. (While this general construction works, I can’t get a satisfactory interpretation in the context of computational model. So I’ll take a detour and interpret a different, but equivalent formulation of Monad in part 4)</p>
<p>Now let’s interpret this using the example of free algebra vs forgetful functors. <script type="math/tex">T</script> simply map any set <script type="math/tex">X</script> to the underlying set of its free algebra <script type="math/tex">F(X)</script>, and maps function between set to the naturally induced map between their corresponding free algebra, as usual. The unit provides the embedding of the original set <script type="math/tex">X</script> into the free algebra(‘s underlying set). The really non-trivial part is the join operation. Notice that <script type="math/tex">T^2X</script> is a set consisting of algebraic expression of algebraic expression over <script type="math/tex">X</script>. If this is befuddling, we can also say that it is an algebraic expression where one layer of variable definition is allowed. For example:
If <script type="math/tex">X = \{ x, y, z \}</script>, then let <script type="math/tex">p = 2x+1, q=x-y+3z, r=y^2 + xz</script>, and let the final expression be <script type="math/tex">p^2 - q + 2r</script>.</p>
<p>Now there is an obvious way to simplify this: just (partially) evaluate it by substitution: <script type="math/tex">(2x+1)^2 - (x-y+3z) +2(y^2 + xz) = \ldots</script></p>
<p>We can see how this is related to the general construction of the join operation: Recalling that an expression evaluator is possible in any algebra (and given by the counit), a substitution evaluator is really just a plain evaluator over the free algebra <script type="math/tex">F(X)</script>, so we have <script type="math/tex">FGF(X) \longrightarrow F(X)</script>. Now simply apply forgetful functor on both sides.</p>
<h2 id="kleisli-category">Kleisli Category</h2>
<p>We use monad in practise instead of adjunction when programming because it is difficult to specify an entirely new computational model, while it is “easy” (in the sense of not having to leave your current programming environment) to implement things within the current computational model. While our conceptual picture is that of constructing/deriving a monad from an adjunction, we can still ask the converse question: does every monad arise from/implies an adjunction?</p>
<p>Happily, it turns out the answer is a resounding “Yes”. In fact there is a family of possible adjunctions that compose to the same monad. Even more luckily, there is both a minimal and maximal such adjunctions, and they are the minimal, <a href="https://en.wikipedia.org/wiki/Kleisli_category">Kleisli category</a> consisting of just the free algebra, versus the maximal, <a href="https://en.wikipedia.org/wiki/F-algebra">Eilenberg-Moore category</a> of all algebra. We will look at both in turns.</p>
<p>Suppose that we just assume the existence of any such adjunction, without knowing what the other category <script type="math/tex">\mathcal{D}</script> is (nor the individual functors <script type="math/tex">F</script> or <script type="math/tex">G</script>). Further assume that they are really just some form of Free algebra/Forgetful functors. At the very least, <script type="math/tex">\mathcal{D}</script> must contain the image of <script type="math/tex">F</script> over <script type="math/tex">\mathcal{C}</script>, which are all the free algebras. Since we don’t really know what <script type="math/tex">F</script> is, we will just let any object <script type="math/tex">X</script> in <script type="math/tex">\mathcal{C}</script> stands for an unknown object <script type="math/tex">F(X)</script>. <sup id="fnref:5"><a href="#fn:5" class="footnote">5</a></sup> Hence we can let the set of objects in <script type="math/tex">\mathcal{D}_{\text{min}}</script> be just all the objects in <script type="math/tex">\mathcal{C}</script>.</p>
<p>Now how about the morphisms? <em>Prima facie</em>, they should really be just the algebra’s homomorphisms, i.e. <script type="math/tex">{\operatorname{Hom}}_{\mathbf{Alg}}(FX, FY)</script>, and composition of morphism is just ordinary composition of function. However we can’t do that since we don’t directly know <script type="math/tex">F</script>. Instead we need to do so indirectly by expressing them in terms of things we have direct access to only. So appealing to the Hom-adjunction property of an adjunction, we have that that set is naturally isomorphic to <script type="math/tex">{\operatorname{Hom}}_{\mathbf{Set}}(X, UFY) = {\operatorname{Hom}}_{\mathbf{Set}}(X, TY)</script>, and we can just let this latter set be the definition of morphisms between <script type="math/tex">X</script> and <script type="math/tex">Y</script> in <script type="math/tex">\mathcal{D}</script>.</p>
<p>A challenge that remains is to define composition of morphisms. Per the reasoning/interpretation for the Hom-Set adjunction we talked about in <a href="/fundamentals/2017/08/28/the-monad-tutorial-fallacy-part-one-introduction-prequel.html">Part 1</a>, given an arrow <script type="math/tex">X \longrightarrow TY</script> it should be possible to uniquely extend it into a morphism <script type="math/tex">TX \longrightarrow TY</script>. Then we can just compose as usual: <script type="math/tex">X \longrightarrow TY \longrightarrow TZ</script> is an arrow from <script type="math/tex">X</script> to <script type="math/tex">TZ</script>, which fits the representation of morphism in <script type="math/tex">\mathcal{D}_{\text{min}}</script>. In programming this extension operation is called a <code class="highlighter-rouge">bind</code> (After switching order of arguments if necessary and using the curried form). <sup id="fnref:6"><a href="#fn:6" class="footnote">6</a></sup></p>
<p>It turns out that bind can be expressed in terms of a Monad by sending <script type="math/tex">X \stackrel{\phi}\longrightarrow TY</script> into <script type="math/tex">TX \stackrel{T\phi}\longrightarrow T^2 Y \stackrel{\mu_Y}\longrightarrow TY</script>. We can prove that this is correct, namely, that the result of extension defined this way, is the same as applying the forgetful functor to the same morphism <script type="math/tex">\phi</script> represented as an honest algebra homomorphism <script type="math/tex">{\operatorname{Hom}}_{\mathbf{Alg}}(FX, FY)</script>.</p>
<p>First note that the extended morphism has at least one <script type="math/tex">T</script> throughout ( <script type="math/tex">UFX \stackrel{UF\phi}\longrightarrow (UF)(UF)Y \stackrel{\mu_Y}\longrightarrow UFY</script>), so we can strip the outer <script type="math/tex">U</script> (due to functor’s law), and then it suffices to show that the morphism <script type="math/tex">FX \stackrel{F\phi}\longrightarrow FUFY \stackrel{\eta_{FY}}\longrightarrow FY</script> is <script type="math/tex">\phi</script> mapped by the natural isomophism between <script type="math/tex">{\operatorname{Hom}}_{\mathbf{Set}}(X, UFY)</script> and <script type="math/tex">{\operatorname{Hom}}_{\mathbf{Alg}}(FX, FY)</script>. Now note that by category theory, the counit <script type="math/tex">\eta_{FY}</script> is really just the identity on <script type="math/tex">UFY</script> mapped through the natural isomorphism, and that by naturality’s commutative diagram, pre-composing it with <script type="math/tex">F\phi</script> is then the same as just pre-composing <script type="math/tex">\text{id}_{UFY}</script> with <script type="math/tex">\phi</script> (which is then still <script type="math/tex">\phi</script>), and then applying the natural isomorphism, and so we are done.</p>
<p>If this is too dense, a rough sketch of the proof is that the counit (expression evaluator) extend to the identity, and that since extension is basically just the natural isomophism of an adjunction, we can indirectly get what we want by applying the counit and relying on the “commutative” property of extension.</p>
<p>For the case of actual free algebra, what this operation amount to is to first induce a map from the free algebras generated by <script type="math/tex">X</script> and <script type="math/tex">TY</script> by considering <script type="math/tex">\phi</script> as just an honest map between sets (ignoring the structures in <script type="math/tex">TY</script>), and then collapse the nested algebraic expression that results by substitution.</p>
<p>Before moving on we want to check that composition defined this way is associative. Since the composition is built out of the extension operator, we only need to check the following:
<script type="math/tex">(g^* \circ f)^* = g^* \circ f^*</script> (Which can be seen as an indirect form of idempotency. It cannot be stated directly because the extension operator cannot be applied to the same function twice)</p>
<p>We compute</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
(g^* \circ f)^* &=& (X \stackrel{f}\longrightarrow TY \stackrel{g^*}\longrightarrow TZ)^* \\
&=& T \left( X \stackrel{f}\longrightarrow TY \stackrel{g^*}\longrightarrow TZ \right) \stackrel{\mu_Z}\longrightarrow TZ \\
&=& TX \stackrel{Tf}\longrightarrow T^2 Y \stackrel{Tg^*}\longrightarrow T^2 Z \stackrel{\mu_Z}\longrightarrow TZ
\end{eqnarray} %]]></script>
<p>Now since <script type="math/tex">g^*</script> is <script type="math/tex">TY \stackrel{Tg}\longrightarrow T^2 Z \stackrel{\mu_Z}\longrightarrow TZ</script>, <script type="math/tex">Tg^*</script> is <script type="math/tex">T^2 Y \stackrel{T^2 g}\longrightarrow T^3 Z \stackrel{T \mu_Z}\longrightarrow T^2 Z</script>.</p>
<p>We can use the associativity of <script type="math/tex">\mu</script>, and then the naturality of <script type="math/tex">\mu</script>, to change to</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{eqnarray}
&& TX \stackrel{Tf}\longrightarrow \left( T^2 Y \stackrel{T^2 g}\longrightarrow T^3 Z \stackrel{T \mu_Z}\longrightarrow T^2 Z \right) \stackrel{\mu_Z}\longrightarrow TZ \\
&=& TX \stackrel{Tf}\longrightarrow T^2 Y \stackrel{T^2 g}\longrightarrow \left( T^3 Z \stackrel{\mu_{TZ}}\longrightarrow T^2 Z \stackrel{\mu_Z}\longrightarrow TZ \right) \\
&=& TX \stackrel{Tf}\longrightarrow \left( T^2 Y \stackrel{\mu_Y}\longrightarrow T Y \stackrel{Tg}\longrightarrow T^2 Z \right) \stackrel{\mu_Z}\longrightarrow TZ \\
&=& g^* \circ f^*
\end{eqnarray} %]]></script>
<p>as desired.</p>
<p>We can also reconstruct <script type="math/tex">F</script> and <script type="math/tex">G</script> from the monad <script type="math/tex">T</script> using Kleisli Category, and it is mostly common sense. <script type="math/tex">F</script> should just send any object in <script type="math/tex">\mathcal{C}</script> to the same object in <script type="math/tex">\mathcal{D}_{\text{min}}</script> (due to the way we relabel things). Morphisms in <script type="math/tex">\mathcal{C}</script> can be embedded in <script type="math/tex">\mathcal{D}_{\text{min}}</script> by just composing with the unit, aka <script type="math/tex">X \stackrel{f}\longrightarrow Y \stackrel{\mu_Y}\longrightarrow TY</script>. (Check that this is a functor!) Similarly, to forget things, <script type="math/tex">X</script> in <script type="math/tex">\mathcal{D}_{\text{min}}</script>, which represent <script type="math/tex">F(X)</script>, should be sent to <script type="math/tex">UF(X) = TX</script>. Morphisms in <script type="math/tex">\mathcal{D}_{\text{min}}</script>, say <script type="math/tex">X \stackrel{g}\longrightarrow TY</script>, will need to sent to some function of type <script type="math/tex">TX \longrightarrow TY</script>. So we just apply the extension operator since it is unique anyway.</p>
<h2 id="eilenberg-moore-category">Eilenberg-Moore Category</h2>
<p>At the opposite extreme, suppose we want to construct the category of all “algebra”. The challenge here is that we need to specify it without referring to anything internal to the objects itself (say, the addition operation in abelian groups), since for the construct to be general we must not make any assumption on what kinds of objects they are actually. Instead we can try to appeal to the interrelationship between objects. This kind of approach to generalizing things beyond their original context/example is known as “categorification” in mathematics.</p>
<p>So an algebra is just a set with all the algebraic operation specified over that set. Per the remarks above we are not allowed to refer to the actual algebra, although we can refer to the underlying sets. (As they are in category <script type="math/tex">\mathcal{C}</script>) Recalling what we said in <a href="/fundamentals/2017/08/28/the-monad-tutorial-fallacy-part-one-introduction-prequel.html">part 1</a>, a cunning trick to recover the algebra is to specify an expression evaluator instead. (This works because expression encompass and generalize algebraic operation. For example, to fully specify <script type="math/tex">+</script>, it suffices to know the value of the expression <script type="math/tex">a + b</script> where <script type="math/tex">a</script> and <script type="math/tex">b</script> can range over everything in <script type="math/tex">X</script>) This we can do: objects in our category <script type="math/tex">\mathcal{D}_\text{max}</script> is just an arrow <script type="math/tex">TX \stackrel{\alpha}\longrightarrow X</script>, where <script type="math/tex">X</script> is in <script type="math/tex">\mathcal{C}</script>. Here <script type="math/tex">X</script> is the underlying set, <script type="math/tex">TX</script> is a free algebra whose elements are all the expressions, and the morphism is then the evaluator (it maps expression over <script type="math/tex">X</script> to value in <script type="math/tex">X</script> again).</p>
<p>However, the formulation here is too loose: we must somehow ensure that the evaluator is valid, that it obey the algebra’s laws/axioms. Part of this work is already done by the free algebra construct hidden within <script type="math/tex">T</script>, since expressions that are provably identical using the axioms of the algebra are quotiented out (i.e. identified) in <script type="math/tex">F(X)</script>. The remaining part is a baggage brought on by the fact that <script type="math/tex">TX</script>, as a free algebra, has extra structures not present when we are just talking about the value of <script type="math/tex">a + b</script> (with <script type="math/tex">a</script> and <script type="math/tex">b</script> mere values and not expressions).</p>
<p>To do this we appeal to the monad, again applying the idea that expression generalize algebraic operation. Namely, we require the following diagram commutes:</p>
<script type="math/tex; mode=display">% <![CDATA[
\array{
T^2 X & \stackrel{T \alpha}\longrightarrow & TX \\
\downarrow^{\mu} & & \downarrow^{\alpha} \\
TX & \stackrel{\alpha}\longrightarrow & X
}
\qquad
\array{
X & \stackrel{\eta}\rightarrow & TX \\
& {}_{\text{id}} \searrow & \downarrow^{\alpha} \\
& & X
} %]]></script>
<p>What does it mean? Suppose we have an expression of expression. The two ways to evaluate it would be to either substitute then evaluate (monad join, followed by <script type="math/tex">\alpha</script>), or evaluate twice - evaluate the value of intermediate variables first, then evaluate the resulting expression. (<script type="math/tex">T \alpha</script> then <script type="math/tex">\alpha</script>) They should obviously be the same. As an example, let <script type="math/tex">u = x + 2y</script> and <script type="math/tex">v = y - z</script>. Suppose that <script type="math/tex">x = 3, y = 7, z = 1</script>. Then <script type="math/tex">u = 17, v = 6</script> and so <script type="math/tex">u + v = 23</script>. On the other hand <script type="math/tex">u + v = x + 3y - z</script>, so the evaluator have to map <script type="math/tex">x + 3y - z</script> to <script type="math/tex">23</script> (as this is not otherwise mandated since this expression is different from either <script type="math/tex">u</script> or <script type="math/tex">v</script>, and <script type="math/tex">TX \longrightarrow X</script> is just a map from set to set).</p>
<p>With the basic structure out of the way, specifying homomorphisms between algebra is actually pretty easy. Consider that a hom is a function between the underlying set which must also respect the algebraic structure. In ordinary algebra this is expressed through commutative diagrams for each algebraic operations. For instance:</p>
<script type="math/tex; mode=display">\require{AMScd}
\begin{CD}
\stackrel{(a, b)}{G \times G} @>+>> \stackrel{(a + b)}{G} \\
@V{(\phi, \phi)}VV @VV{\phi}V \\
\underset{(\phi(a), \phi(b))}{H \times H} @>+>> \underset{\phi(a + b) = \phi(a) + \phi(b)}{H}
\end{CD}</script>
<p>Because the structure of each algebra is already constrained by the requirement above, we can just naively generalize this diagram, turning the artificial <script type="math/tex">G \times G</script> into the more general set of expression <script type="math/tex">TX</script>:</p>
<script type="math/tex; mode=display">\require{AMScd}
\begin{CD}
T(X) @>{\alpha}>> X \\
@V{Tf}VV @VVfV \\
T(Y) @>{\beta}>> Y
\end{CD}</script>
<p>It remains to show that this does indeed form a category, and the main work to do is to check the composition of morphisms. Well, we can just stack the commutative diagram together:</p>
<script type="math/tex; mode=display">\require{AMScd}
\begin{CD}
T(X) @>{\alpha}>> X \\
@V{Tf}VV @VVfV \\
T(Y) @>{\beta}>> Y \\
@V{Tg}VV @VVgV \\
T(Z) @>{\gamma}>> Z \\
\end{CD}</script>
<p>Recalling that our goal is to prove the outer square commutes, we can perform diagram chasing: first note that the arrows <script type="math/tex">X \stackrel{f}\longrightarrow Y \stackrel{g}\longrightarrow Z</script> is the same as <script type="math/tex">X \stackrel{g \circ f}\longrightarrow Z</script>, and by functor’s axiom we can apply <script type="math/tex">T</script> to this “equation” to get that the arrow on the left hand side can also be decomposed: <script type="math/tex">X \stackrel{T(g \circ f)}\longrightarrow Z</script> is the same as <script type="math/tex">X \stackrel{Tf}\longrightarrow Y \stackrel{Tg}\longrightarrow Z</script>. Then just apply the commutativity condition for the two inner squares to transform:</p>
<script type="math/tex; mode=display">(g \circ f) \circ \alpha = g \circ (f \circ \alpha) = g \circ \beta \circ Tf = \gamma \circ Tg \circ Tf = \gamma \circ T(g \circ f)</script>
<h2 id="full-circle-back">Full Circle Back</h2>
<p>Finally, as free algebra are also algebra, we want to see how the Kleisli category embed into the Eilenberg Moore Category. Well, the underlying set of a free algebra generated by <script type="math/tex">X</script> is just <script type="math/tex">TX</script>, so we need an arrow <script type="math/tex">T^2 X \longrightarrow TX</script>. The monad join fits the bill here (and not just because the signature match<sup id="fnref:7"><a href="#fn:7" class="footnote">7</a></sup>) Why? Because algebraic operations in a free algebra is in fact defined by the substitution semantics. Anyway we still need to verify the conditions/diagrams for being an algebra. The first diagram is commutative because <script type="math/tex">\mu</script> is a natural transform. The second diagram is just the unit law of monad.</p>
<p>How about morphism? We need a representation that is of type <script type="math/tex">TX \longrightarrow TY</script> - which means for a morphism <script type="math/tex">\phi: X \longrightarrow TY</script> we should use its extension <script type="math/tex">\phi^*</script>. Then we should check that it is in fact a morphism in the Eilenberg-Moore Category:</p>
<script type="math/tex; mode=display">% <![CDATA[
\array{
T^2 X & \stackrel{\mu}\longrightarrow & TX \\
\downarrow^{T^2 \phi} & & \downarrow^{T \phi} \\
T^3 Y & \stackrel{\mu T}{\cdots\cdots} & T^2 Y \\
\downarrow^{T \mu} & & \downarrow^{\mu} \\
T^2 Y & \stackrel{\mu}\longrightarrow & TY
} %]]></script>
<p>The diagram constructed from <script type="math/tex">\phi^*</script> is shown above (with the obvious step for decomposing <script type="math/tex">T(\phi^*)</script> omitted as it is the same as what we’ve seen before - just use the functor’s law). To prove that it commutes, connect the middle row with the morphism <script type="math/tex">\mu T</script>. Then the bottom square commutes by the multiplication law of monad, while the upper square commutes due to <script type="math/tex">\mu</script> being a natural transform. Then we’re done by performing the usual diagram chasing.</p>
<p>Incidentally, this provide an alternative proof that the extension operator is correct: as extension is unique, we only need to ensure that:</p>
<ol>
<li>The extended morphism remains the same on <script type="math/tex">X</script></li>
<li>It is a homomorphism on <script type="math/tex">\mathcal{D}</script> and not just <script type="math/tex">\mathcal{C}</script>.</li>
</ol>
<p>The argument above amount to showing 2. For 1, we precompose with the unit:
<script type="math/tex">% <![CDATA[
\begin{eqnarray}
&&X \stackrel{\eta}\longrightarrow TX \stackrel{T \phi}\longrightarrow T^2 Y \stackrel{\mu}\longrightarrow TY \\
&=& X \stackrel{\phi}\longrightarrow TY \stackrel{\eta}\longrightarrow T^2 Y \stackrel{\mu}\longrightarrow TY \text{(unit is natural)} \\
&=& X \stackrel{\phi}\longrightarrow TY.
\end{eqnarray} %]]></script></p>
<h2 id="one-more-thing">One more thing…</h2>
<p>Right when I’m doing the final editing I come across <a href="http://blog.sigfpe.com/2009/12/where-do-monads-come-from.html">this</a> and at a glance the approach it took looks similar to what I did here (free algebra/algebraic expression, that kind of stuff), although I took a pure math approach in case you’re too thick :P.</p>
<p><em>(Monad Series: To be continued…)</em></p>
<hr />
<div class="footnotes">
<ol>
<li id="fn:1">
<p>This does not mean that the adjunction construct is not algebraic - in fact a Category can be thought of as a groupoid, where elements can be multiplied provided their domain/co-domain match. Then an ordinary group is just a groupoid with only one domain so that anything can be multiplied with anything. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>But I can’t resist the temptation to include a Matrix quote:</p>
<blockquote>
<p>The Matrix is everywhere. It is all around us. Even now, in this very room. You can see it when you look out your window or when you turn on your television. You can feel it when you go to work… when you go to church… when you pay your taxes. It is the world that has been pulled over your eyes to blind you from the truth.</p>
<p>…</p>
<p>Unfortunately, no one can be…told what the Matrix is. You have to see it for yourself.</p>
<p>Morpheus, The Matrix (1999)</p>
</blockquote>
<p><a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>But fear not, for the Haskell community recognized this problem long ago (How can they not when you have endless waves of people asking about Monad?), and I think somewhere, nice and smart people are trying to find a better way. For example, see <a href="https://www.slideshare.net/linecorp/the-monad-fear">this presentation</a>. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
<li id="fn:4">
<p>But Haskell also has something similar - template. See <a href="https://www.reddit.com/r/haskell/comments/61r64w/what_does_the_free_monad_offer_that_macros_dont/">this</a> for an advocacy. <a href="#fnref:4" class="reversefootnote">↩</a></p>
</li>
<li id="fn:5">
<p>Those not coming from a math/abstract algebra background may find this strange. Rest assured it is a culturally standard practise in math. One of the very first thing you learn in university algebra is that the actual name/label given to things in a set doesn’t matter - so long as you always get the right things when called upon to. (In fact recall that the definition of a Category doesn’t really require looking into the <em>content</em> of an object) So using <script type="math/tex">X</script> instead of the actual <script type="math/tex">F(X)</script> doesn’t hurt - given such a label we can get back the real things, metaphorically, by applying the functor <script type="math/tex">F</script>. <a href="#fnref:5" class="reversefootnote">↩</a></p>
</li>
<li id="fn:6">
<p>The naming here may look strange. It will look much more “natural” (no pun intended) when doing actual programming where we care about actually applying those functions. <a href="#fnref:6" class="reversefootnote">↩</a></p>
</li>
<li id="fn:7">
<p>A gripe I have with the statically typed camp of programming, even for the sufficiently-advanced-type-that-offer-automatic-safety-guarantee, is that placing too much focus on the type risks crowding out my cognitive capacity to understand the actual semantics of a piece of code. It takes discipline to resist the temptation to consider a function correct just because the signature match. <a href="#fnref:7" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>(Go to Part 1 of this series) In part 2 of this series we explain the technical core of the theory of Monad. Of central importance is the attempt to provide algebraic interpretation of everything, as well as to convert things into suitably algebraic structures. We will also revisit this issue at an arguably “better” (but more advanced) angle in part 5. While the picture for an adjunction is relatively clear, it suffers from needing to work with two categories at the same time. Monad fixes this by composing the left and right adjoint functors into an endofunctor , over a single category. An additional advantage with this construction is that it is more algebraic as it is now composable: we can iterate to get , etc for example. 1 This does not mean that the adjunction construct is not algebraic - in fact a Category can be thought of as a groupoid, where elements can be multiplied provided their domain/co-domain match. Then an ordinary group is just a groupoid with only one domain so that anything can be multiplied with anything. ↩A Quick Note on the Clojure (Backend) Web Space2017-09-14T20:16:00+08:002017-09-14T20:16:00+08:00https://lemonteaa.github.io/misc/2017/09/14/a-quick-note-on-the-clojure-backend-web-space<p><em>(Last Updated on 16 Sep 2017. Thanks nha, gsnewmark and danielcompton for corrections and discussions!)</em></p>
<p><em>(Note to Experts: if you find any factual error in this note, please tell me by leaving a comment in the comment section below. Many thanks!)</em></p>
<h2 id="frameworks">Frameworks</h2>
<p>Clojure as a community has emphasized two points (among others):</p>
<ol>
<li>No complicated, Rube-Goldberg-Machine framework - prefer simple, focused, composable libraries for specific features. Pick your own choice for each feature and combine them in your application.</li>
<li>Data is a first-class-citizen. Instead of coupling it with function, let it stand alone and have functions operating on/transforming them.</li>
</ol>
<p>That being said it is tedious and intimidating for a rookie to have to make informed choice on every step. So there is still some “frameworks”:
<!--more--></p>
<ul>
<li><a href="http://www.luminusweb.net/">Luminus</a> is a funky thing: while it is called a “micro framework” on its webpage, if you look closer you will find nothing beyond the series of guide webpages. It is not even a library! Instead it is just a lein template you can call to scaffold a web project with sensible default choices of libraries (per point 1 above) and all the tedious configuration done for you.</li>
<li><a href="http://arachne-framework.org/">Arachne</a> (still in alpha) is more opinionated and co-opt point 2 above: the app is data-driven and the kind of messy configurations so common in other web framework (in other languages) are “real data” and given first-class, systematic/unified treatment. (Similar project and/or successor on this line is the <a href="https://github.com/duct-framework/duct">Duct</a> framework)</li>
<li><a href="http://clojure-liberator.github.io/liberator/">Liberator</a> is more a library targeted to those who want to write a RESTful web service backend. It has put heavy emphasis on fully embracing the doctrine of REST - even those that are less adhered to in practise (<a href="http://timelessrepo.com/haters-gonna-hateoas">HATEOAS</a> comes to mind). It seems that its more pragmatic, <em>asynchronous</em> counterpart - <a href="http://pedestal.io/">Pedestal</a> - has more adoption in the real world.
<ul>
<li>Continuing along the same line, <a href="https://github.com/juxt/yada">yada</a> combine characteristics from both framework above: it provide comprehensive supports for HTTP semantics, <em>and</em> is asynchronous. <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup> Notes however that at this moment it only supports Aleph (with plan to supports other in the future).</li>
</ul>
</li>
</ul>
<h2 id="build-tools">Build Tools</h2>
<p>Just in case you don’t know (yet), there are two:</p>
<ul>
<li><a href="https://leiningen.org/">Leiningen</a> (lein in short) is the standard one and is a task runner, declarative project configuration, and dependency management combined (sort of like Maven if you’re in Java, npm if you’re from Javascript, rake if you’re in Ruby I think(?), XX if you’re from language YY - really most modern languages nowadays have this particular combination as the default)</li>
<li><a href="http://boot-clj.com/">Boot</a> is (ironically) the new kid on the block and pursue a procedural style that allows more flexibility/programmability. (E.g. Just Ant for Java, or Gulp over Grunt for Javascript guy - I know, that two are ancient in 2017, but still)</li>
</ul>
<h2 id="servers">Servers</h2>
<p><em>(I assume you have the Java background to understand, well, the Java part. If not, well that’s too bad - wait until my planned next post where I will briefly talk about that in a tangential subsection :P )</em></p>
<p>Of course two “universal” deployment options are:</p>
<ul>
<li>Build an uberjar that is completely standalone (web application is packaged with an embedded web server) and then just execute it on the JVM</li>
<li>Build an (Servlet-compliant) uberwar and deploy it into any Java web server / web container</li>
</ul>
<p>Unfortunately thing starts to get unruly at this point, so tighten your seat belt…</p>
<p>While Servlets are a good thing <em>in spite of</em> how it might be <a href="http://misko.hevery.com/2009/04/08/how-to-do-everything-wrong-with-servlets/">badly designed</a>, by abstracting away differences between web/application servers and presenting a uniform interface to application developer, maintaining this kind of standard (especially in Java) pretty much ensure that it will be slow to pick up on innovation in the web server space. But the rise of asynchronous web servers such as Node.js and other emerging forms of interaction (such as websockets) put on pressure for adoption. When the standard fails to adapt <em>fast enough</em> (Newer version of the Servlet spec has support for them), people will begin to work around (and hence without) it. So instead of a unified interface, expects custom API specific to each web server.</p>
<p>Another trend that may be unfamiliar to people used to working in an enterprise context is the rise of embedded server. <sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup> It is interesting to note a kind of <em>Inversion of Control</em> here: instead of deploying an application to the container/server and letting the server’s execution manage starting/stopping the app, we now have explicit control: the application contains the server and triggers the start/stop of the server through application logic. <sup id="fnref:3"><a href="#fn:3" class="footnote">3</a></sup> One advantage of this approach is that the <em>implicit</em> logic of a server managing a deployment is now <a href="https://www.python.org/dev/peps/pep-0020/"><em>explicit</em></a> so that when problem occurs, things are more traceable.</p>
<p>So, with these understanding, what clojure-specific web servers do we have?</p>
<p>First, <a href="https://github.com/ring-clojure/ring">ring</a> is the Clojure analogue of the Servlet spec. It is <em>a HTTP server abstraction/interface plus a number of small libraries</em>.</p>
<ul>
<li><a href="http://www.http-kit.org/">http-kit</a> is a ring-compatible web server written in Java and Clojure.</li>
<li>Jetty and Netty are classical Java web servers. Jetty is a standard, embedded Java server, while Netty is closer to Node.js with emphasis on the Transport Layer by providing access to TCP/UDP/socket server as well as supporting asynchronity.
<ul>
<li>Ring itself has an adaptor for Jetty, and that’s an (easy) default choice in some contexts.</li>
<li><a href="http://aleph.io/">Aleph</a> is a server that’s a thin wrapper over Netty. It can also acts as drop-in replacement for any ring-compliant server.</li>
</ul>
</li>
<li><a href="http://immutant.org/">Immutant</a> is a more heavyweight library/server that includes other services needed for more sophisticated system (messaging queue for instance). It can be deployed to either Java’s WildFly or since 2.x the <a href="http://undertow.io/">Undertow</a> web server.</li>
</ul>
<h2 id="libraries">Libraries</h2>
<p>There are just too many of them to discuss here. Instead I just highlight some of the foundational ones:</p>
<ul>
<li>For something similar to dependency injection in Java, we’ve got application lifecycle/state management libraries: <a href="https://github.com/stuartsierra/component">Component</a> vs <a href="https://github.com/tolitius/mount">Mount</a>.</li>
<li>For the routing part of your typical/classical MVC framework, we’ve got <a href="https://github.com/stuartsierra/component">Compojure</a> vs just plain <a href="https://github.com/ring-clojure">ring</a> (Remember ring is an API + a collection of libraries) vs <a href="https://github.com/juxt/bidi">bidi</a>.</li>
</ul>
<h2 id="and-one-more-thing">And One More Thing…</h2>
<p>I haven’t mentioned database in this post so far. Well of course you can use any as long as JDBC supports it… database is not usually tied to a particular programming language. But since we’re talking about clojure, it is criminal not to mention <a href="http://www.datomic.com/">Datomic</a>, a <a href="http://augustl.com/blog/2016/datomic_the_most_innovative_db_youve_never_heard_of/">super-innovative</a> “database” made by <a href="https://cognitect.com/">Cognitect</a> (if it can be called that, per the talks by Rick on the <a href="https://www.infoq.com/presentations/Datomic-Database-Value">Database-as-a-value</a>). Unfortunately, it is one of the rare infrastructural piece of software in our ecosystem that is not Open Source nor free (they have a free version, but it has <a href="https://my.datomic.com/downloads/free">restrictions</a>).</p>
<h2 id="references">References</h2>
<p>Thanks Google! (and the people in the Clojure community ;) )</p>
<ul>
<li><a href="https://groups.google.com/forum/#!topic/clojure/9nsVazn44u0">Announcing Aleph</a>
<ul>
<li><a href="https://news.ycombinator.com/item?id=1498198">Discussion on HN</a></li>
</ul>
</li>
<li><a href="https://www.booleanknot.com/blog/2017/05/09/advancing-duct.html">Advancing Duct</a></li>
<li><a href="https://adambard.com/blog/i-finally-get-boot/">My first time using boot over leiningen</a></li>
<li><a href="https://stackoverflow.com/questions/5385407/whats-the-difference-between-jetty-and-netty">Differences between Jetty vs Netty</a></li>
<li><a href="https://stackoverflow.com/questions/167262/how-do-you-make-a-web-application-in-clojure/32165654#32165654">Discussion on Clojure web servers on stackoverflow</a></li>
</ul>
<hr />
<div class="footnotes">
<ol>
<li id="fn:1">
<p>See their <a href="https://github.com/juxt/yada/blob/master/dev/resources/comparison-guide.md">wiki</a> for their own assesement/comparison of yada vs Liberator and Pedestal. Also see their <a href="https://juxt.pro/blog/posts/yada-1.html">blog post</a> for a sales pitch. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>My own pet theory: Ever heard of DevOps? Instead of externalising config and dependent systems with a dedicated, separate team to manage them, why not put everything back to the hands of developer with the same powerful set of tool - general purpose programming language - to manage config? Really nice for lone wolf developer… the operation department, not so much. <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
<li id="fn:3">
<p>I will confess to having personal prejudice that embedded server are somehow slower: no they are not <em>a priori</em> slow. How could it be? Afterall web/application server are just codes, and where exactly are those code executed does not affect its performance. <a href="#fnref:3" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>(Last Updated on 16 Sep 2017. Thanks nha, gsnewmark and danielcompton for corrections and discussions!) (Note to Experts: if you find any factual error in this note, please tell me by leaving a comment in the comment section below. Many thanks!) Frameworks Clojure as a community has emphasized two points (among others): No complicated, Rube-Goldberg-Machine framework - prefer simple, focused, composable libraries for specific features. Pick your own choice for each feature and combine them in your application. Data is a first-class-citizen. Instead of coupling it with function, let it stand alone and have functions operating on/transforming them. That being said it is tedious and intimidating for a rookie to have to make informed choice on every step. So there is still some “frameworks”:The Monad Tutorial Fallacy, Part One: Introduction/Prequel2017-08-28T22:56:00+08:002017-08-28T22:56:00+08:00https://lemonteaa.github.io/fundamentals/2017/08/28/the-monad-tutorial-fallacy-part-one-introduction-prequel<p>Understanding how monad works is a kind of mandatory rite of passage for any programmer aspiring to truly master the functional paradigm (of the advanced statically typed kind). Unfortunately it is an unforgiving trial: in spite of the large amount of tutorials out there the construction itself just seem to have some kind of irreducible complexity whose reason is mostly opaque to neophytes.</p>
<p>Interestingly, although I mainly studied math when I was an undergrad, I first came to learn monad through the usual programmer’s way - see some hand-wavy explanation of what it is/does, see introductory examples of monads, read the abstract interface, read (and try to understand) more examples, struggle, then “think” that I understood it, realize (a few months to a year later) that I haven’t - that I have missed some facet of it, rinse and repeat, despair at some point that I am probably never going to truly master it…
<!--more--></p>
<p>Until I study monad from the perspective of a pure mathematician with no thought given to applications. Then everything clicked. “Why don’t you tell me this earlier?”</p>
<p>So as a service mainly to myself, I’m writing this series of post retracing the story of monad (cleaned up of course). <strong><em>In clear violation of the warning of the infamous <a href="https://byorgey.wordpress.com/2009/01/12/abstraction-intuition-and-the-monad-tutorial-fallacy/">Monad Tutorial Fallacy</a>.</em></strong> You have been warned.</p>
<p>In retrospect, the reasons that learning-through-example/pattern didn’t work for me is because:</p>
<ol>
<li>The pattern emerging from the examples is not the monad itself (at least, not its interface nor its implementation). It is sort of like the mismatch of thinking in the problem domain vs the solution domain.</li>
<li>Unlike interface in traditional OOP languages, there are variants of the monad interface(s) with translation “formula” between them - a situation where Haskell typeclass shines but which add confusion to beginners. More importantly, unlike other context where things decompose obviously, the “moving pieces” of the monad interface seems to be tightly coupled to each other and look like the wrong decomposition, so that while it is possible to say understand any of its parts if I focus my mind on it for a while, it is hard to hold a mental conception/model of the whole thing in my head in one piece. Exactly the same situation where understanding a large, interlocking system is scary because you can’t really hold the full picture in your head and have to work on it piece-by-piece, always worrying about possible side effects.</li>
</ol>
<p>Okay, let’s begin the story. <em>(Prerequisite: <del>Bachelor’s degree in Pure Math</del> Knowledge of some category theory and basic abstract algebra)</em></p>
<p><em>(Warning to actual mathematician: skipping all proofs as this is a tech blog and not a math blog, and because you can find them elsewhere/do it yourself anyway. Or is it just a lame excuse on my part to not make this post ridiculously long?)</em></p>
<p>Before Monad, there’s the Adjoint Functor. But even that is a pretty mysterious abstraction as impenetrable as the monad. So let’s talk about <del>concrete</del> prototypical example, which in our story will be the free algebra.</p>
<p>Recall that given any unstructured set <script type="math/tex">X</script>, the free algebra generated by <script type="math/tex">X</script>/over <script type="math/tex">X</script>, denoted <script type="math/tex">F(X)</script>, is the algebra formed by taking all formal expressions over <script type="math/tex">X</script> with the obvious algebraic operations, and where elements are identified if and only if they can be proved to be identities under the axioms of the algebraic system we are using. (So <script type="math/tex">a+(b+c)</script> is the same element as <script type="math/tex">(a+b)+c</script>.) The intuition for this construction is that it is a tautological one: the formal expressions represent things that simply have to exists in any algebra containing <script type="math/tex">X</script> (e.g. If <script type="math/tex">a, b</script> are in <script type="math/tex">X</script>, and we have some algebra <script type="math/tex">A</script> containing <script type="math/tex">X</script>, then necessarily <script type="math/tex">a + b</script> is in <script type="math/tex">A</script> by closure law), and then we impose the minimal constraint on the generated algebra by identifying only what have to be (those that can be proved to be equivalent).</p>
<p>Now the categorical way to think of free algebra is that they are a universal construction: There is an obvious embedding <script type="math/tex">\iota</script> of <script type="math/tex">X</script> into <script type="math/tex">F(X)</script>, and for any other algebra <script type="math/tex">Y</script> that contains <script type="math/tex">X</script> (through the embedding <script type="math/tex">g</script>), we always have an unique homomorphism <script type="math/tex">\phi: F(X) \rightarrow Y</script> so that the diagram below commutes:</p>
<script type="math/tex; mode=display">% <![CDATA[
\array{
X & \stackrel{\iota}\rightarrow & F(X) \\
& {}_g \searrow & \downarrow^\phi \\
& & Y
} %]]></script>
<p>If you actually read category theory textbook you may find the above diagram doesn’t look quite right: the problem is that we have been a bit loose and didn’t clearly distinguish which object belongs in which category: we can’t really just say <script type="math/tex">X \rightarrow F(X)</script> as <script type="math/tex">X</script> is in the category of set, while <script type="math/tex">F(X)</script> is in the category of algebra. We can fix this (at the cost of making the diagram more complicated looking) by using the forgetful functor <script type="math/tex">G: \mathbf{Algebra} \rightarrow \mathbf{Set}</script>, which just stripe away all the algebraic operations, leaving only the set of elements. So the embedding become: <script type="math/tex">I: X \rightarrow GF(X)</script> as a map/morphism between sets. And voila - this is the natural transformation called <strong>the unit</strong>.</p>
<p>There is also a nifty idea that will become critical later on: adjoint functor is a theory of duality, and whenever there is such an adjunction the properties outlined above also work if we flip all the arrows (i.e. take the dual in category-speak). In particular there would naturally be a morphism <script type="math/tex">E: FG(A) \rightarrow A</script> for any algebra <script type="math/tex">A</script>, called <strong>the counit</strong>. But what is this? <script type="math/tex">FG(A)</script> is the free algebra whose underlying elements are <script type="math/tex">G(A)</script>, or just the elements of <script type="math/tex">A</script>. So <script type="math/tex">FG(A)</script>’s elements are formal expressions, where all those variables take values in <script type="math/tex">A</script>, <em>where algebraic operations are well defined since <script type="math/tex">A</script> is an algebra</em>. <strong>E is an expression evaluator!</strong></p>
<p>But there’s still one thing more (at least). The duality/symmetry of an adjunction become apparent once we express it in terms of a <em>natural</em> isomorphism between two hom-sets. (In fact this representation is the key ingredient in the proof of the duality mentioned above anyway) Packaged up in a nice formula, it reads: <script type="math/tex">{\operatorname{Hom}}_{\mathbf{Algebra}}(FX, Y) \cong {\operatorname{Hom}}_{\mathbf{Set}}(X, GY)</script>. What does it mean? Well, similar to how any linear transformation is uniquely determined by the values it takes on any basis of the source vector space, any homomorphism from the free algebra <script type="math/tex">F(X)</script> to any other algebra <script type="math/tex">Y</script>, is uniquely determined by the value it takes over a natural basis, which is just the underlying set/set of variables <script type="math/tex">X</script>. Equivalently, any such map induces a full homomorphism (and not just a map of sets) over <script type="math/tex">F(X)</script> by the obvious algebra law (e.g. If <script type="math/tex">\phi(a) = p</script> and <script type="math/tex">\phi(b) = q</script>, then <script type="math/tex">\phi(a + 2b)</script> can be defined as <script type="math/tex">p + 2q</script>). The reason for the appearance of the forgetful functor <script type="math/tex">G</script> is the same as in the discussion on universal property.</p>
<p>Now we mostly understand Adjoint Functors (excluding the more theory side of things, such as the Special and General Adjoint Functor Theorem for existence/construction of <script type="math/tex">G</script> from <script type="math/tex">F</script>), but so what? Here comes the important part: <strong><em>we can interpret (typed) pure functional programming through category theory</em></strong>, and a suitably comprehension interpretive framework serves as the foundation for understanding monad, among other things.</p>
<p>In the simplest situation, a Category is a typed version of the composition higher order function. The objects in the Category are the type of the system, morphisms from <script type="math/tex">A</script> to <script type="math/tex">B</script> are the pure (computable) functions of type <script type="math/tex">A \rightarrow B</script>, and composition of arrows (in the category) correspond to function compositions (in our programming model). Even though it appears to be trivial and useless, this naive model can be extended and changed in various way, and you may even specify your own Category/Programming Model! (I promise to give examples of these in Part 3 or something of this series)</p>
<p><em>Adjoint Functors represents the duality between an unstructured, raw computational model/domain versus a structured, contextual computational model/domain.</em> So we first design and specify a custom-made Category <script type="math/tex">\mathcal{D}</script> (where all its element - object, morphism and composition - need not be the standard/usual one) that capture/model the particular context we want, and Category <script type="math/tex">\mathcal{C}</script> remains the standard one in last paragraph. Then the functors <script type="math/tex">F: \mathcal{C} \rightarrow \mathcal{D}</script> and <script type="math/tex">G: \mathcal{C} \leftarrow \mathcal{D}</script> provides the natural translation between these two contrasting model of computation. In particular <script type="math/tex">F</script> is an embedding functor: any computation can be trivially considered as computation with context by simply carrying out all computation without using or depending on the context at all. (In practice though describing <script type="math/tex">F</script> is not trivial since category <script type="math/tex">\mathcal{D}</script> is itself not trivial) On the other hand, any computation with context is still a computation (Church-Turing thesis), so <script type="math/tex">U</script> interpret it as plain computation by forgetting/stripping away the extra structure (but no information is lost - it is the marking of context, and not the content of the context, that is being stripped). In more silly terms, <script type="math/tex">F</script> tells you that you can technically write assembly in any high level (Turing complete) language, while <script type="math/tex">U</script> tells you that ultimately any such high level language will be compiled down into assembly anyway.</p>Understanding how monad works is a kind of mandatory rite of passage for any programmer aspiring to truly master the functional paradigm (of the advanced statically typed kind). Unfortunately it is an unforgiving trial: in spite of the large amount of tutorials out there the construction itself just seem to have some kind of irreducible complexity whose reason is mostly opaque to neophytes. Interestingly, although I mainly studied math when I was an undergrad, I first came to learn monad through the usual programmer’s way - see some hand-wavy explanation of what it is/does, see introductory examples of monads, read the abstract interface, read (and try to understand) more examples, struggle, then “think” that I understood it, realize (a few months to a year later) that I haven’t - that I have missed some facet of it, rinse and repeat, despair at some point that I am probably never going to truly master it…Micro Web Framework in Python2017-08-27T20:26:00+08:002017-08-27T20:26:00+08:00https://lemonteaa.github.io/exercises/2017/08/27/micro-web-framework-in-python<p>Web Framework is arguably the most often used kind of library in real world, as well as being something that keeps being reinvented for every programming language and every generation of programmer. Given such ubiquity, it is strange that it is not included in the canon of the undergrad Computer Science cirriculum alongside Operating System, Database, and Compiler. This post seeks to bridge this gap by presenting a series of short exercise that culiminates in building your own micro web framework. Have fun!</p>
<p><em>Important Note:</em> This exercise is inspired by <a href="https://ruslanspivak.com/lsbaws-part1/">Ruslan’s “Let’s Build A Web Server”</a> series as a kind of follow up. As we will be using online resources to help, please read the reference section at the end first before beginning the exercise.
<!--more--></p>
<h2 id="0-preparation">0. Preparation</h2>
<p><em>(Difficulty: Easy to Medium+ depending on how serious you get)</em></p>
<p>First we want to refine some aspect of the codes in the blog series to make further development on top of them more pleasant to work with.</p>
<p>a) You may come across problem when trying to test part 1 using telnet in window. It turns out that telnet may decide to send packets as it receives characters you type, rather than sending them all at once. Since the function <code class="highlighter-rouge">client_connection.recv(1024)</code> means trying to get <strong>something</strong> with an upper bound of fetching 1024 bytes at most, you may get cut off after entering a single character. What we really need is some sort of protocol coupled with a stream interface - we should be persistantly reading until we know enough about the beginning part of the content sent to know the actual length of the whole payload - in this case it is the <code class="highlighter-rouge">Content-Length</code> http header field that will tell us.</p>
<p>Extract suitable function from part 2’s code to help with parsing the packet’s content, and then write a code snippet to properly read the packet and insert it into the code in part 1. (Hint: use the readline and read function)</p>
<p>b) Add logging to the code in part 1 and part 2 so that we can debug more easily. After importing logging, add the line</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="n">logging</span><span class="o">.</span><span class="n">basicConfig</span><span class="p">(</span><span class="n">stream</span><span class="o">=</span><span class="n">sys</span><span class="o">.</span><span class="n">stdout</span><span class="p">,</span> <span class="n">level</span><span class="o">=</span><span class="n">logging</span><span class="o">.</span><span class="n">DEBUG</span><span class="p">,</span> <span class="n">format</span><span class="o">=</span><span class="s">'</span><span class="si">%(asctime)</span><span class="s">s [</span><span class="si">%(levelname)</span><span class="s">s] </span><span class="si">%(message)</span><span class="s">s'</span><span class="p">)</span></code></pre></figure>
<p>for a minimal setup, then add log in places you think are important.</p>
<p>c) The code in part 2 skipped some details of the WSGI requirement to simplify things, and we need to add some of that back in.
i. The <code class="highlighter-rouge">PATH_INFO</code> parameter is currently the whole path. Write functions to extract query string into <code class="highlighter-rouge">QUERY_STRING</code> and correct the line for <code class="highlighter-rouge">PATH_INFO</code> correspondingly.
ii. Write functions to extract all http headers and add them to the environ dict in one stroke. (Be careful about special case of <code class="highlighter-rouge">CONTENT_LENGTH</code>)</p>
<p>(Optional) d) Study using multiprocessing to make a httpd daemon so that starting/stopping server is less tedious.</p>
<h2 id="1-elements-of-a-framework">1. Elements of a Framework</h2>
<p>A Web Framework provides its own model of how things work, as well as a more convenient and/or more powerful interface to the user than the actual interface to the web server (which in this case is the WSGI). They achieve this by layering on top of the basic interface, doing some of the tasks that can be generalized. We will build a heavily simplified micro web framework incrementally, starting with miscellaneous functions.</p>
<p><em>(Difficulty: Easy)</em></p>
<p>a) The first issue is that the data structure in WSGI is too low-level with everything packed in one single env dict - we want to present the data in a cleaner way that is also more meaningful from a user’s point of view.</p>
<p>Let’s create a class named <code class="highlighter-rouge">HttpReq</code> (Shortform for HTTP Request) that is just data structure:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">HttpReq</span><span class="p">:</span>
<span class="k">pass</span></code></pre></figure>
<p>It should have the following attributes:</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>url</td>
<td>The URL being requested.</td>
</tr>
<tr>
<td>method</td>
<td><code class="highlighter-rouge">GET</code> or <code class="highlighter-rouge">POST</code></td>
</tr>
<tr>
<td>headers</td>
<td>a python dict whose keys are the header names and the values are the corresponding header values.</td>
</tr>
<tr>
<td>getparams</td>
<td>if request method is <code class="highlighter-rouge">GET</code>, this should be a python dict of the request parameters whose keys are parameter names and the values are the corresponding parameter values. The value should be a python array if there is a repeat of parameter name.</td>
</tr>
<tr>
<td>postparams</td>
<td>if request method is <code class="highlighter-rouge">POST</code>, this should be a python dict with the request parameters in the same schema as getparams.</td>
</tr>
</tbody>
</table>
<p>i. Write a helper function <code class="highlighter-rouge">extractHeaders(env)</code> to extract headers from the env dict according to the requirement for the class <code class="highlighter-rouge">HttpReq</code>.</p>
<p>ii. Write a function <code class="highlighter-rouge">get_req(env)</code> to construct and return a <code class="highlighter-rouge">HttpReq</code> object from the env dict, conforming to the above specification.</p>
<p><em>Note:</em> You are allowed to use the module <code class="highlighter-rouge">urlparse</code> (<code class="highlighter-rouge">urllib.parse</code> if using Python 3) in standard library for this question.</p>
<p><em>(Difficulty: Hard)</em></p>
<p>b) The second and third (and fourth) issues are that it is still too cumbersome to work directly with HTTP request and response (too much boilerplate code to extract data from request object manually as well as assembly/building response object), and for larger projects with many ‘pages’ located at various URL, doing everything in one function is confusing and difficult to manage (try searching through/jumping around the code!). Different frameworks may use different mechanisms, but some common ideas are:</p>
<ul>
<li>Use handler that focus on dealing with requests on one particular (or one family of) resource.</li>
<li>Inject various kinds of parameters into the handler, and only require the handler to return the core content of the response (or return an abstracted response object)</li>
<li>Some frameworks will also allow automatic configuration of the web application itselfs to varying degree.</li>
</ul>
<p>In this exercise, we will implement a ‘router’ in the framework (explained in question 2 below), use dependency injection-like method to provide the parameters, and use decorator for auto-config (a similar thing is called annotation-driven config in Java).</p>
<p>Implement a decorator called <code class="highlighter-rouge">route</code> that is applied to methods (which is a kind of callable in python) acting as handler. The decorator should have the following arguments:</p>
<table>
<thead>
<tr>
<th>Name</th>
<th>Required?</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td>url</td>
<td><em>mandatory</em></td>
<td>for which URL will this handler be able to process the request?</td>
</tr>
<tr>
<td>method</td>
<td><em>optional</em></td>
<td>any restriction on request type? <code class="highlighter-rouge">GET</code> if handler only deals with GET request, <code class="highlighter-rouge">POST</code> for POST request only, and <code class="highlighter-rouge">ANY</code> for no restriction. Defaults to <code class="highlighter-rouge">ANY</code>.</td>
</tr>
</tbody>
</table>
<p>It should wrap the method and apply dependency injection by inspecting the method argument names and providing the corresponding request parameters (using <code class="highlighter-rouge">None</code> as value if it doesn’t exist).</p>
<p>As an example:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">TodoApp</span><span class="p">:</span>
<span class="nd">@route</span><span class="p">(</span><span class="s">'/todo/create'</span><span class="p">,</span> <span class="n">method</span><span class="o">=</span><span class="s">'POST'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">addTodoListItem</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">task</span><span class="p">,</span> <span class="n">pos</span><span class="p">):</span>
<span class="c"># ...snip...</span></code></pre></figure>
<p>Here the route’s URL is ‘/todo/create’, the method is <code class="highlighter-rouge">POST</code>, and the decorator should extract request parameters <code class="highlighter-rouge">task</code> and <code class="highlighter-rouge">pos</code> if they exists.</p>
<h2 id="2-piecing-stuff-together">2. Piecing Stuff Together</h2>
<p>A router is something that route a request to a suitable handler for further processing. The handlers are matched/tested against the request by examining its HTTP method and URL, and in more advanced cases patterns/regex can be used when matching URL (which we won’t do here). Usually a map/registry of handlers are internally recorded and used for matching.</p>
<p><em>(Difficulty: Easy+)</em></p>
<p>a) In older generations of web framework the user is responsible for providing/configuring the handler map manually/explicitly, for example by calling a function with the full handler map as argument, or by calling a register function for each handler/route. Newer generation can be auto-config as mentioned in Q1 above.</p>
<p>i. Write a function <code class="highlighter-rouge">buildHandlerMap(clz)</code>, that accept a class meta-object as argument, extract all handlers inside that class (i.e. methods in the class decorated with <code class="highlighter-rouge">@route</code>), and return the handler map. The map should be a list of 4-tuple with format <code class="highlighter-rouge">(<method name>, <URL in route decorator>, <method in route decorator>, <the handler callable>)</code>.</p>
<p>(Hint: modify your code in Q1 b to add suitable attributes to the callable, so that the meta-data stored in the decorator itself can be inspected)</p>
<p>ii. Write a function <code class="highlighter-rouge">dispatchReq(handlerMap, req)</code>, that when given the handler map built from Q2 a(i), and a <code class="highlighter-rouge">HttpReq</code> object, will perform the routing as described above. It should call the first handler in the list that matches successfully with req supplied as argument, and then return the result of that call, or raise an Exception with message <code class="highlighter-rouge">No route found</code> otherwise.</p>
<p><em>(Difficulty: Medium)</em></p>
<p>b) We are now finally ready to write the ‘main function’ of our framework. Finish the following code snippet:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">class</span> <span class="nc">microFrame</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">clz</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">handlerMap</span> <span class="o">=</span> <span class="n">buildHandlerMap</span><span class="p">(</span><span class="n">clz</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">app</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">env</span><span class="p">,</span> <span class="n">start_response</span><span class="p">):</span>
<span class="c"># TODO</span></code></pre></figure>
<p><em>(Difficulty: Easy+)</em></p>
<p>c) i. Now test the framework you’ve just written by writting an example web application. Your test should include at least one route each for:</p>
<ul>
<li>Returning a html form to submit</li>
<li>A page accepting query parameter called through GET</li>
<li>A page that accept form submit through POST</li>
</ul>
<p>You may optionally test for default argument in function in Python.</p>
<p>ii. Test drive your web application. Open an interactive Python session, and type suitable command to package a web application, “deploy” it to the custom web server (after enhancements in question 0), and start the server. Open your browser and tests that it behaves as expected.</p>
<h2 id="optional-3-extension">(Optional) 3. Extension</h2>
<p>In this question we try to extend our web framework to support more functionality. We will provide only the goal - you are free to come up with your own idea! (See reference section for explanation, supporting materials etc if you get stuck)</p>
<p>a) Add support for manipulating Cookie.</p>
<p>b) i. Based on part a, now implement Session.</p>
<p>ii. Can you make your implementation of a and b(i) be thread-safe?</p>
<h2 id="reference">Reference</h2>
<h3 id="q0">Q.0</h3>
<p>The original tutorial/DIY series is at <a href="https://ruslanspivak.com/lsbaws-part1/">here</a> (Part 1, follow links at bottom to get to other parts), and we assume that the reader has already read (but not neccessarily worked out) both part 1 and part 2 throughout this exercise.</p>
<p>We will be working and perhaps tinklering with both HTTP’s protocol detail and drill down on WSGI’s interface a bit. For http, the authoritative reference is the <a href="https://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4">original spec</a>.</p>
<p>Python’s standard socket library provide a function <code class="highlighter-rouge">socket.makefile</code> to allow accessing socket through a file-like api. However there are some subtlety involved and it have (previously?) some limitations. Nonetheless it is good enough for our exercise. See <a href="http://stackoverflow.com/questions/12203800/should-i-close-makefileed-socket-and-it-is-original-socket-independently">this</a> for hints on how to use it.</p>
<p>One of Python’s strength has been in its battery included philosophy while providing an easy, accessible interface and supporting a wide variety of system integration tasks. Logging is taken care of in a similar spirit and it is in fact a built in library. Read <a href="https://docs.python.org/2/howto/logging.html#logging-basic-tutorial">Python’s doc</a> for an introduction.</p>
<p>It turns out that you cannot test the simple server in part 2 with chrome because of speculative connection that send nothing, see <a href="http://stackoverflow.com/questions/4761913/server-socket-receives-2-http-requests-when-i-send-from-chrome-and-receives-one">this</a> for explanation.</p>
<p>WSGI is an old school interface that still works - similar to the CGI interface at the dawn of web 2.0 one or two decades before. Refer to <a href="https://wsgi.readthedocs.io/en/latest/definitions.html">the WSGI Community Website</a> for a brief description of the environ keys, and read <a href="http://wsgi.tutorial.codepoint.net/">Clodoaldo Neto’s WSGI Tutorial</a> to quickily learn how to work with it practically.</p>
<p>For a concise but still in-depth explanation of the multiprocessing module, see <a href="https://pymotw.com/2/multiprocessing/">the Python Module Of the Week’s entry</a>. This website is also useful in general for getting a tour of various Python modules.</p>
<p>Aside from specific pages mentioned above, Python has a good amount of online materials ranging from beginner-level-tutorial to more advanced articles. One example is <a href="http://docs.python-guide.org/en/latest/">The Hitchhiker’s Guide to Python by Kenneth Reitz</a>. Meanwhile, do not forget that Python’s <a href="https://docs.python.org/2/index.html">official documentation</a> has reasonable average quality too, so check it out before looking for supplementary materials.</p>
<h3 id="q1">Q.1</h3>
<p>Web framework is a foundational, classical piece of software and as such there has been numerous specific frameworks, in a wide variety of context (e.g. different language) throughout the years. In spite of “recent” innovations such as Ruby On Rails and Express in Node.js (and many new generation, lightweight frameworks emphasizing interactive development), their basic principles remain the same (and so in my opinion belongs to the realm of Computer Science). See <a href="https://developer.mozilla.org/en-US/docs/Learn/Server-side/First_steps/Web_frameworks">the MDN Web Docs’ entry on Server-side Web Framework</a> for an introduction to what they do. (<em>Sidenote:</em> The MDN is also in general an authoritative reference especially for front-end developer)</p>
<p>Decorator is a somewhat tricky feature in Python because it uses higher order function in an essential way, and also because there are different use cases with subtly different syntax. Two blog posts that explain these points are <a href="https://www.thecodeship.com/patterns/guide-to-python-function-decorators/">[1]</a> at a beginner level, and <a href="http://scottlobdell.me/2015/04/decorators-arguments-python/">[2]</a> that also covers the more advanced cases of using a class as decorator.</p>
<h3 id="q3">Q.3</h3>
<p>Cookie and web sessions are closely related concepts. Cookie solves the problem of maintaining state across HTTP requests (the HTTP protocol itself is stateless) by changing the protocol specification to require browser cooperation. Server can set a designated header in its HTTP response, which the browser should honor by saving the data in its own memory and repeating the data in all subsequent requests (until expiry or reset) in another designated header. See <a href="https://en.wikipedia.org/wiki/HTTP_cookie">Wikipedia’s Article</a> for details and <a href="https://tools.ietf.org/html/rfc6265">RFC6265</a> for the specification itself.</p>
<p>Web Session solve a similar class of problem as Cookie, but with the state stored on server side. The usual way to do it is to build on top of Cookie: the server generate a session ID and set it as the Cookie value. Any state that need to be saved is stored on server side by associating that data with the session ID, which can later be retrieved by lookup. Authentication can be achieved by choosing a suitable scheme for generating the session ID with various cryptographic/security properties (Full security requires many more considerations though). In this exercise you can ignore security issues and just use the <code class="highlighter-rouge">uuid</code> module in Python. See <a href="http://machinesaredigging.com/2013/10/29/how-does-a-web-session-work/">[3]</a> for an illustrated example. Also see <a href="https://web.stanford.edu/~ouster/cgi-bin/cs142-fall10/lecture.php?topic=cookie">[4]</a> for a condensed summary of the discussions above.</p>
<p>Concurrency is an intrinsically difficult problem that we sometimes cannot avoid (espcially more so in this age when the free lunch given by Mooer’s law is basically over). Python offer a number of facilities for doing multithreading - see for example <a href="http://www.laurentluce.com/posts/python-threads-synchronization-locks-rlocks-semaphores-conditions-events-and-queues/comment-page-1/">[5]</a> for a catalogue of options.</p>
<p>Unfortunately it also turns out that concurrency is one of the weak spot in Python due to the infamous Global Interpreter Lock (GIL). In short, even if multi-threading is done correctly you may not get the desired performance boost. See <a href="https://jeffknupp.com/blog/2012/03/31/pythons-hardest-problem/">Jeff Knupp’s blog post “Python’s Hardest Problem”</a> for an account of the problem and its <a href="https://jeffknupp.com/blog/2013/06/30/pythons-hardest-problem-revisited/">follow up article</a> for a list of possible remedies. The standard advise given when asking this problem on forums such as stackoverflow is “Use multiprocessing instead of threading”. Read <a href="https://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python">this</a> for a summarized comparison.</p>
<p><em>(Wow, you really read to the end! As a reward for that, <a href="https://github.com/lemonteaa/python-exercise/tree/master/micro_web_framework">here</a> is my own work-out of these exercises - exluding bonus sections)</em></p>Web Framework is arguably the most often used kind of library in real world, as well as being something that keeps being reinvented for every programming language and every generation of programmer. Given such ubiquity, it is strange that it is not included in the canon of the undergrad Computer Science cirriculum alongside Operating System, Database, and Compiler. This post seeks to bridge this gap by presenting a series of short exercise that culiminates in building your own micro web framework. Have fun! Important Note: This exercise is inspired by Ruslan’s “Let’s Build A Web Server” series as a kind of follow up. As we will be using online resources to help, please read the reference section at the end first before beginning the exercise.