1 1. Execute forward and backward modes of neural network by hand or by using Python (your choice) for 3 iterations (epochs). The model is shown in Fig. 1. This network comprises of a hidden layer with 3 ReLU units and a squared-error loss. (Note: Use tanh as activation function in the output unit and ReLU for hidden layer) (40 points) ReLU (x) = Jx, if x 20 10, otherwise (Hint: Notice that this function is non-differentiable around 0. Therefore, when computing your gradients, you will have to compute one gradient if the input is larger than 0, another if it is smaller than 0, and ignore when the input is 0 and thus non- differentiable.) Step 1: Initialize the weights to these random values shown below [W11 W21 W31] [0.60 0.70 0.00] W12 W22 W32 0.01 0.43 0.88 = W11 [0.02] W12 0.03 W13 0.09 The data is provided in the following matrix where samples are stored in rows. 0.75 0.80 0.20 0.05 -0.75 0.80 0.20 -0.05 Their corresponding labels are y = [1, 1, −1, −1]. Step 2: You may use the recursive patterns to update parameters of this network. HINT: Pseudo-code for manual calculations: 1. Forward the input and record (i) input to the unit, s;, and (ii) output of the unit, z, for each unit j in all layers. 2. Compute the loss and record it. Use L 0.5 (yout - y)² 3. Compute the error signal, Sout= მC at the output. asout 4. Propagate Sout backwards to compute 8; = 3 at hidden units. მაქ S 2 W₁₁ X(1) X(2) W 31 W 21 W12 W22 W 32 Yout W12 W13 Fig. 1. NN topology Sout 5. Compute gradients (i.e. AW, Aw) using these and update weights using gradient descent. Hint: W(+1) W (t) - a AW 6. Repeat 3 times. Problem 1 Instructions: All hand solutions MUST contain the details including formulas of pseudocode (Steps 1- 6, for 3 iterations) you are using and not only the final values.