Now, what the Vikings spoke was Proto-Norse. Several significant changes took place around AD 900, after which we start talking about Old Norse. Written scraps of Proto-Norse are extremely rare, so most of the grammatical forms and pronunciation have to be recreated using the tools of historical linguistics.
The writing system for pre-conversion Old Norse is what we call the Younger Futhark. This was a simplified system of runes that emerged about the time Proto-Norse was turning into Old Norse. Before that, there was a fuller set of runes we call the Elder Futhark, which were developed to write Proto-Germanic, the language that preceded Proto-Norse. The Elder Futhark was probably developed from a variant of the Greek Alphabet that was in use before Greek was standardized during the Athenian ascendancy. It was probably taken north and adapted by some other trading partner, before being adopted and adapted by the Germanic people yet further on. So, the runes probably entered Germanic society some time in the last half millennium BC, before Germanic began to break up into separate Eastern (Gothic), Western (Ingvaeonic [that's us], Istvaeonic, Irminonic), and Northern (Proto-Norse) languages. (There was an Old English adaptation of the runic alphabet, too, called the futhorc.) After the introduction of Latin literacy by way of Christian monks, the Latin alphabet was used to write the Germanic languages, though several special characters remained (like þ) to express sounds that didn't appear in Latin.
Hat tip to Jackson Crawford, Old Norse specialist, whose YouTube videos filled in some of the details I was hazy about.