When he states "get thee to a nunnery" to Ophelia, he is expressing pent-up anger towards his mother, who he feels has been unfaithful and incestuous when she married his uncle. At the beginning of the play itself, we see a brooding Hamlet who seems almost more upset by his mother's marriage than by his father's death.
