Yellow area would have been the city walls in the time of Nehemiah.
Herod rebuilt this wall and also built a wall around the New City.
Later he built a third wall increasing the size of the City of Jerusalem.  At this time he connected his new city wall to the preexisting temple mount wall, further fortifying the Mount area.  That explains why there is no northern wall of the Temple mount we see today.

The ruins of the oldest east gate still remain under the east gate of today.  Nehemiah called this older gate in Hezekiah's wall the Commaders Gate, which as we can see was located at the north end of the east wall, but after Herod extended the east wall on both the north and the south ends then the gate in the east wall isn't fully at the north end anymore.  The problem is the Temple and compound was there is Nehmeiah's day when the walled in part of the Mount was much smaller, and the Temple wasn't built this far to the north on the Mount for this east gate to be the Golden gate which entered directly onto Solomon's stoa/portico.  In fact the Meah tower was at that place which later was increased in size drmatically and called Fort Antonia.