Is speaking the most difficult skill? In my opinion it is hard to say. It depends on the age of the speaker, his level of English, how long is he learning and also what kind of English does he operates (possess). There is a big difference between a student of a university where he is studying English and a person (e.g. at the same age) who knows English from some courses. So that is why I am not sure about question whether is speaking the most difficult skill. From my own experience I would say that speaking is maybe at the same level as listening. But I think that listening is little bit more difficult than speaking because when you are speaking, you have some time to thing about the vocabulary, to choose the right word and what is the most important, if you are not able to find the right word (or you do not know how to say it in English) you can use circumlocution, describe the word but also the listener can help you in some way. Whereas in listening there are not so many possibilities how to understand the meaning. There are not so many ways how to help yourself.
Mr. Brown says that ,,spoken language does indeed contain a good deal of redundancy, it also has many reduced forms…” That is why (in my opinion) is listening more difficult. Person who is on the elementary or intermediate level is not able to understand reduced forms in the stream of sounds (speech) like I’ll /ail/, you’d /youd/ etc. He does not understand what does “ail” mean. He knows only “I will”. But when speaker say ,,I will” not “I’ll” there is no mistake.
Another example why speaking is not the most difficult (while listening is) is because of ,,the rephrasings, repetitions, elaborations and little insertions of ´I mean´ and ´you know´ etc.” (Brown). This means that using of these “empty” phrases is not obligatory for the speaker. Of course the speech is more natural by using these phrases. On the other hand, these “empty” phrases may cause to listener-beginner some problems in understanding although sometimes they can help to be fixed (oriented) in given information.
There is also another reason why listening is more difficult that speaking. Let us take the speed of the speech. Many people who were abroad (Great Britain, USA…) say that they had problems with listening comprehension for first one or two months. Every native has his own speed of speaking (which is sometimes quite fast) and I am not speaking about dialect or accent. Person who is listening to native can have problems to catch some word or even meaning. But in speaking it is easier. It does not matter which kind of speed you are speaking or which kind of accent do you have. Native or non-native will understand what are you talking about
At the end I would say that the most important is that we can not separate listening from speaking and speaking from listening. Both are difficult (more than reading and writing) but according to my opinion and my experience, speaking is not the most difficult skill. I think that listening is the most difficult.