\sum
produces the summation operator \sum; \Sigma
produces a capital sigma \Sigma.
You haven't used the correct z component of \mathbf{f}; it should be z^2, not z as you have.
As regards the rest of your working, it is simplest to just use x and y as parameters rather than introducing u and v. Then d\boldsymbol{\sigma} =<br />
\begin{pmatrix} 1 \\ 0 \\ \frac{\partial z}{\partial x} \end{pmatrix} \times<br />
\begin{pmatrix} 0 \\ 1 \\ \frac{\partial z}{\partial y} \end{pmatrix}\,dx\,dy<br />
= \begin{pmatrix} - \frac{\partial z}{\partial x} \\ -\frac{\partial z}{\partial y} \\ 1 \end{pmatrix}\,dx\,dy<br />
= \begin{pmatrix} 3 \\ \tfrac32 \\ 1\end{pmatrix}\,dx\,dy<br /> and the surface integral then reduces to <br />
\begin{split}<br />
\iint_\Sigma \mathbf{f} \cdot d\boldsymbol{\sigma} &=<br />
\int_0^1 \int_0^{2(1-x)} 3x^2 + \tfrac32xy + \left(3 - 3x - \tfrac32y\right)^2\,dy \,dx<br />
\\<br />
&= \int_0^1 \int_0^{2(1-x)} \tfrac32 x(2x+y) + \left(3 - \tfrac32(2x+y)\right)^2\,dy\,dx \\<br />
&= \int_0^1 \int_{2x}^2 \tfrac32 xu + 2\left(3 - \tfrac32 u\right)^2\,du\,dx<br />
\end{split}<br /> using the substitution u = 2x + y in the inner integral (I think this substitution simplifies the algebra significantly, thereby reducing the scope for errors).