The 21-year old, who scored 12 goals in 18 games last season in the Swiss top-flight, arrived in England after Zurich agreed a reported fee of £10 million.
"Brighton & Hove Albion have withdrawn from the potential transfer of Raphael Dwamena, after the player failed the club's medical," a club's statement read.
"Albion had previously agreed terms with both Dwamena and FC Zurich over a potential deal for the 21-year-old, which would have been subject to a UK work permit.
"The club would like to wish Raphael the very best for the future, as he returns to FC Zurich," the statement added.
Dwamena made his debut for Ghana in June against Ethiopia (5-0) in the 2018 Africa Cup of Nations qualifiers, scoring a brace.
He would've become the sixth Ghanaian in the Premier League this season after the Ayew brothers, Jeffery Schlupp, Christian Atsu and Daniel Amartey.