Max Value of $b$ for Real Roots of $f(x)$ and $g(x)$

[anemone]

Let $a$ and $b$ be real numbers and $r,\,s$ and $t$ be the roots of $f(x)=x^3+ax^2+bx-1$ and $g(x)=x^3+mx^2+nx+p$ has roots $r^2,\,s^2$ and $t^2$. If $g(-1)=-5$, find the maximum possible value of $b$.

 

[anemone]

Spoiler: Hint

Rewrite $g(x)$ in terms of $a$ and $b$...

 

[anemone]

Spoiler: Solution of other

From Vieta's formulas, we know

$=a=r+s+t,\,b=rs+st+tr,\,1=rst$

and

$m=-(r^2+s^2+t^2),\,n=r^2s^2+s^2t^2+t^2r^2,\,p=-r^2s^2t^2$

Let's try to express $m,\,n$ and $p$ in terms of $a$ and $b$. The easiest one is $p$:
$p=-(rst)^2=-1$

From $m$, we square $r+s+t$:
$a^2=(r+s+t)^2=r^2+s^2+t^2+2(rs+st+tr)=-m+2b\implies m=2b-a^2$

Finally, for $n$ we can square $rs+st+tr$:
$b^2=(rs+st+tr)^2=r^2s^2+s^2t^2+t^2r^2+2(r^2st+rs^2t+t^2rs)=n+2rst(r+s+t)=n-2a\implies n=b^2+2a$

So now we write $g(x)$ in terms of $a$ and $b$:
$g(x)=x^3+(2b-a^2)x^2+(b^2+2a)x-1$

We know that $g(-1)=-5$; when we plug this into our equation for $g(x)$ we get

$-5=(-1)^3+(2b-a^2)(-1)^2+(b^2+2a)(-1)-1=1b-a^2-b^2-2a-2\implies a^2+2a+b^2-2b-3=0$

We seek the largest possible value of $b$. Since $a$ is real, we know that the discriminant of this quadratic must be non-negative. In particular,

$2^2-4(b^2-2b-3)\ge 0\implies b^2-2b-4\le 0$

Solving this quadratic gives us that the largest possible value of $b$ is $1+\sqrt{5}$.