$\because \left(x+3\right)^{2}\geqslant 0$，$|y-4|\geqslant 0$，
$\therefore$当式子$\left(x+3\right)^{2}+|y-4|+2$取最小值时，$x+3=0$，$y-4=0$，
解得$x=-3$，$y=4$，
$\therefore x^{y}=\left(-3\right)^{4}=81$.
故答案为：$81$.

$\because \left(x+3\right)^{2}\geqslant 0$，$|y-4|\geqslant 0$，
$\therefore$当式子$\left(x+3\right)^{2}+|y-4|+2$取最小值时，$x+3=0$，$y-4=0$，
解得$x=-3$，$y=4$，
$\therefore x^{y}=\left(-3\right)^{4}=81$.
故答案为：$81$.