The Lionesses fell behind in front of a record-breaking 77,768 crowd at Wembley after just nine minutes of action , with Alexandra Popp heading home for the visitors.
Ellen White's one-touch strike levelled the match just before half-time but Germany had the final say when Klara Buhl slid the ball past England goalkeeper Mary Earps in the 90th minute.
Since finishing fourth at the World Cup in France earlier this year, Neville's side have only picked up one win from five, suffering narrow losses against Norway and Brazil, while also recording a draw with Belgium and a win over Portugal.
The 42-year-old head coach defended his players after the match, with it his belief that recent results show he's "not good enough" currently to take England's women forward.
"I've got to take responsibilities for those results, the buck has got to start and finish with me," Neville told a post-match press conference.
"The team reflects the manager and the team aren't good enough [right now] which means I'm not good enough."
When asked if England's level had dropped since their return from France, Neville added: "There is no hiding from that fact. One in seven is totally unacceptable. It is not what I see as success and we need to do something about it very quickly.
"We can talk about the emotional highs about the World Cup, but when you get into November that becomes an excuse."
The Lionesses must pick themselves up quickly for a trip to the Czech Republic for another friendly on Tuesday, which takes on greater significance after their latest defeat.
Another loss would increase pressure on Neville significantly and he acknowledges it will be a "massive test" for England to get back on track against another strong opponent.
"They've got to go again on Tuesday," he added. "It's a challenge, it's my job to make sure that they're in the right frame of mind and to pick the right team on Tuesday.
"To get the right result in the Czech Republic will be a massive test for us now in terms of how our players can respond to disappointments.
"You think about the games in tournaments that we've got coming up. Games in a short space of time happened in the Olympics, they happened in the Euros.
"We've got to recover quickly. And we've got to make sure that we produce."