Some of Veeck's most memorable publicity stunts occurred during his tenure with the Browns, including the appearance on August 19, 1951, by little person Eddie Gaedel. Veeck sent Gaedel to pinch hit in the bottom of the first of the game. Wearing elf like shoes and "1/8" as his uniform number, Gaedel was walked on four straight pitches and then was pulled for a pinch runner. Shortly afterwards "Grandstand Manager's Day" – involving Veeck, Connie Mack, and thousands of regular fans, enabled the crowd to vote on various in-game strategic decisions by holding up placards: the Browns won, 5–3, snapping a four-game losing streak.
After the 1952 season, Veeck suggested that the American League clubs share radio and television revenue with visiting clubs. Outvoted, he refused to allow the Browns' opponents to broadcast games played against his team on the road. The league responded by eliminating the lucrative Friday night games in St. Louis. A year later Cardinal owner Fred Saigh was convicted of tax evasion. Facing certain banishment from baseball, he was forced to put the Cardinals up for sale. Most of the bids came from out-of-town interests, and it appeared that Veeck would succeed in driving the Cardinals out of town. However Saigh accepted a much lower bid from St. Louis-based brewing giant Anheuser-Busch, who entered the picture with the specific intent of keeping the Cardinals in town. Veeck quickly realized that the Cardinals now had more resources than he could possibly hope to match. Reluctantly, he decided to leave St. Louis and find another place to play. As a preliminary step, he sold Sportsman's Park to the Cardinals. Veeck would have probably had to sell it in any event; the 44-year old park was in a poor state of repair, and even with the rent from the Cardinals he did not have the money to bring it up to code.
At first Veeck considered moving the Browns back to Milwaukee (where they had played their inaugural season in 1901). Milwaukee used recently-built Milwaukee County Stadium in an attempt to entice the Browns. However, the decision was in the hands of the Boston Braves. For the Browns to move, the minor league Brewers would be shut down. The Braves wanted another team with the same talent, and an agreement was not made in time for opening day. Ironically, a few weeks later, the Braves themselves moved to Milwaukee. St. Louis was known to want the team to stay, so some in St. Louis campaigned for the removal of Veeck. He then got in touch with a group that was looking to bring a Major League franchise to Baltimore. After the 1953 season, Veeck agreed in principle to sell half his stock to Baltimore attorney Clarence Miles, the leader of the Baltimore group, and his other partners. He would have remained the principal owner, with approximately a 40 percent interest. Even though league president Will Harridge told him approval was certain, only four owners—two short of the necessary six for passage—supported it. Realizing that the other owners simply wanted him out of the picture (indeed, he was facing threats of having his franchise canceled), Veeck agreed to sell his entire stake to Miles' group, who then moved the Browns to Baltimore as the Orioles.