oagre, your line " the reasons do not make sense to us now and are utterly unconvincing" pretty much nails it, when talking about the whys, in America's entry into Vietnam. To understand this, one must look at the overall picture and that was the fact that the west and in particular the US, were embroiled in the Cold War and Communism was the enemy.
Although the US didn't have time for the old Colonial powers and their grip on their former colonies, it was of necessity that they back them and turn a blind eye because of the threat of a Communist takeover of these old colonies. Vietnam was one such place and although the French were doing the fighting, they were backed by American money and materiel to do so and whenever the French wanted to pull out, it was the American's pushing them to stay in. In the final stages of the war in Vietnam for the French, during the battle of Dien Bien Phu, there were a number of "men in power" at the top levels of government in the US who advocated and pushed for the use of nuclear weapons, on behalf of the French, to save them and win the war. Eisenhower, to his credit stood his ground and wouldn't hear of using nuclear weapons on Asiatic peoples once again in the span of a less than a decade of doing so on Japan.
After this Vietnam was separated, much like Korea, but without major powers having forces ready and available. The Communists or Viet Minh, kept infiltrating back into the south and although the American's didn't have combat troops there, they did have military advisers and over time, the American's started to lose some of these advisers. They escalated their presence and there became the need to protect their interests in Vietnam and in the opinion of many a scholar, the same type of "power elite" guided the President into sending actual American combat troops into war in Vietnam, by manufacturing the evidence to do so. The Gulf of Tonkin incident is one such happening that has many holes in it.
This war in Vietnam was exactly what the hawks wanted and although it wasn't a showdown with the Russians, it would have to do. Since before the end of WW2, there were those in the Allied camp who didn't trust the Russians and by the end off that conflict thought they may as well just keep rolling along and fight the Russians, while they still had the men in uniform and the materiel at hand. So Vietnam would have to do for this and the "power elite" convinced the president that American military might would prevail and like many from days long ago, thought they'd do so in days, rather than years. We know what the outcome was to this thought and the escalation in number of troops as well.