(Note: This is a short preparatory post for the real one, where I plan to share my experience dabbling with the rusty but beautiful keras-rl library. I will release that after I finally fulfill my promise and deliver the PPO+A2C integration that has been dragged longer than I am comfortable with )

(Hint: some of the not-explicitly-referenced links are just Wiki, but many others are hidden treasure troves. Thanks for all these authors for their amazing works! )

Modern Artificial Intelligence is a paradoxical case of an unstoppable force meeting an unmovable object. On one hand, this field has been so hyped in the last few years that some people worry about a bubble and potential backlash (probably bad memories from the last AI winter ?); on the other hand its progress is undeniable even to hardcore skeptics - even if we assume that there will be no more new breakthrough (which is quite unlikely), fully exploiting/applying the technology available right now to its potential will still take a long long time (if nothing else, because the list of new applications opening up from this round of new methods is huuuge ). This is an interesting time indeed.

So, if you’re here, you probably wants to get into the field, but just like any bandwagon, the signal-to-noise ratio isn’t exactly nice. How do I gain a foothold and stay?

I personally believe that principles matter. To borrow some armchair buddhism, things in this world are impermanent, always in flux - whatever libraries, frameworks, or even language you hold dear has no guarantee to stay alive, much less the same, as the industry moves forward (or in circle?) But ideas live on, and may even reincarnate when the time is right.

Which is why I’m now outlining a list of the enduring “basics”, to answer this question.

To make things simple, it boils down to these few things:

Fundamental

• Learn programming, preferably Python (because of its strong ecosystem, mainstream status, and beginner-friendliness)
• Learn some scientific computing (the usual barrage of panda, matplotlib, numpy etc for Python), specifically functions/syntax for working with vector/matrix/tensor ¹ - i.e. vectorization (See this for the guide in matlab), and also try out some symbolic computing (The Wolfram code for that is f[x_] := x^4 * Sin[x] then f'[x], see the doc)
• Learn math (good if undergrad major level in both pure and applied maths, best if PhD in selected areas )
  • Linear algebra (the neural interconnections are expressed by matrix, and convolution is an important example of a linear operator ²)
  • Calculus (to help do manual computation related to the quantified performance of a network)
  • Probability theory/Statistic/Stochastic process (The main trend of modern AI is statistical in nature³ - Instead of analyzing any particular training/testing data, we treat the whole ensemble/possible inputs seem in real life statistically, which is much more tractable. Stochastic process, which is roughly speaking the more eloquent brother of probability theory, is useful for more complicated setup like in reinforcement learning, where the basic framework is already Markov Decision Process)
  • Optimization algorithm/theory (The whole Neural Network thing is just optimizing a function’s fit to data by tuning parameters to minimize error while constraining the complexity of our function, to be deliberately provocative. Anyway, the first algorithm you will see is the Stochastic Gradient Descent, which comes from the simple and naive gradient descent, but have some not-so-obvious new behavior.)

Concepts in Machine Learning

Once you’re done with that, or at least have a good enough understanding, go and learn basic concepts in machine learning:

• The data-centric, or statistical paradigm,
• supervised vs (un/semi)-supervised learning,
• the basic workflow,
• feature engineering,
• choosing algorithm/methods,
• overfitting/generalization problem,
• pitfalls in testing ⁴ /cross validation

To list a few.

After that, it is really no different from any other learning:

Search, read, think/tinker, repeat

The end. (See? It’s not that hard afterall )

Actually, not yet… to help you along (and to thank you for bearing with me for a supposedly “short note”), here is a filtered list of links/resources I find helpful:

References

For a comprehensive introduction to all these diverse ideas, the e-book written by Leonardo Araujo dos Santos seems to do the tricks.

For understanding neural network, check out this book. It’s a bit long, but very worth it as a long term investment. If you’re short on time/prefer a more hands-on approach, then see Stanford’s basic tutorial.

We will be skipping all those baroque network architecture as that’s not our focus today.

For reinforcement learning, the resources are scattered:

• Andrej Karpathy’s blog post is a popular introductory post that’s nonetheless deep in some aspects.

• Ruben Fiszel’s blog post - He is an intern who learnt massively in that short timespan and as a result has a trove of treasure/insights/experience to share

More formally, there are some tutorial series that gradually ease in people to successively more complex concepts: this, and this.

In short, learn about the setup of reinforcement learning, then learn about the theory of MDP, then learn about Q-function/value-based approach, then policy-based method, then the actor-critic architecture.

Congratulations, you’re now ready!

---