A couple of things should be noted at the start of this post: 
 

First, downtown Manhattan is not all that large an area, so there are lots of different locations of historic significance within a block of each other.
 

Second, downtown Manhattan has been settled for well over 350 years and there are still a lot of historic sites still standing (or memorials to those that aren't).
 

So, let's start our wandering at Pier A, which is the southernmost tip of the island and also the point closest to both the Statue of Liberty and Ellis Island (second image).
 

Pier A
 

Ellis Island
 

A fairly short walk from there, you can visit Wall Street (but I didn't get a decent picture of the Stock Exchange) and then a block or so later, to Federal Hall, which was where the US government convened and where George Washington was inaugurated as President.

Federal Hall and Trinity Church
 

On the far left of the above photos, you can see Trinity Church, which dates back to the 1700s (the version seen here was rebuilt in 1846 (after a fire destroyed the original in 1776 and the second was damaged by weather).  Wander around there and you will quickly see the grave/memorial below (and many others I didn't know to stop and look for):
 

Associated with Trinity is St. Paul's Chapel, which has another historic, picturesque graveyard.
 

And then, across the street from St. Paul's Chapel is where the twin towers of World Trade Center once stood.  The 9/11 memorial is still under construction, but the footprints of the two towers have been excavated and converted into a very touching site.
 

9/11 Memorial
 

9/11 Memorial
 

No, I do not know the person named or who put the flower there.  It does add to the sense of the place, though.
 

Finally, next door is where the new Freedom Tower looms over the downtown skyline at 1,776 feet tall.
 

Freedom Tower
 

Freedom Tower
 

Freedom Tower
 

The first two images of the Tower were taken at the base, looking up to get a sense of the height.  The final one is a stitched panorama taken by the South Tower pool at the 9/11 Memorial.