Epoch An epoch means put the whole training set into network for training once. . Batch A whole dataset might be too large for us to compute. So we divide it into several batches, and put batches into the network. This also indicates that we may update the network for multiple times in a single epoch. . Iteration Interation number indicates how many times we need to put a dataset into the network....

By Yinggan XU Dibbla This is generated by a previous courses (not included in Lee’s 2022 series), video can be found: CNN The motivation is that we can of course use MLP to find a function such that we do image classification etc. However, it’s not necessary and not efficient due to the tremendous number of parameters. We are going to use the properties of images themselves. Before that, we need to know the structure of a picture....

By Yinggan XU Dibbla The tutorial video can be found here This notebook will only cover the basic optimizers and their ideas. However, the optimizers for DL remains a very interesting question. Background Knowledge $\mu$-strong Convexity We can refer to this note. A function $f$ is $\mu$-strong convex if: $$f(y)\ge f(x)+\nabla f(x)^T(y-x) + \frac{\mu}{2}||y-x||^2\newline \text{for some $\mu\ge0$ and for all $x,y$}$$ Note that Strong convexity doesn’t necessarily require the function to be differentiable, and the gradient is replaced by the sub-gradient when the function is non-smooth....

Environment PATH & .bashrc Add a dir to PATH Add this line to .bashrc if your dir is “/home/yinggan/.local/bin”: export PATH="/home/yinggan/.local/bin:$PATH" Disk Check disk status df -h Check file disk usage du --max-depth=1 -h [file position] You can search further for du command. ls -lh This also works. Python Environment VSC install tensorboard extension The VSC auto run this command /bin/python3 /home/yinggan/.vscode-server/extensions/ms-python.python-2022.10.1/pythonFiles/shell_exec.py /bin/python3 -m pip install -U tensorboard /tmp/tmp-3690113vmd9xrwV8cpD.log And we also have warnings:...

By Yinggan XU Dibbla In this Lecture, Lee introduces the idea of select best model through validation set performance. Lee also explains how deep NN outperforms fat (wide) NN. Validation set The CORE question we want to figure out is Why I used validation set but still overfit? graph LR id1[model 1 with para space H1] id2[model 2 with para space H2] id3[model 3 with para space H3] id4[Validation Set] id1-->id4 id2-->id4 id3-->id4 id5[Validation-Loss-1=0....