For what it's worth, I spent my gestational months and early years in what amounted to a construction site of an apartment. It was in a NYC building that was getting a full renovation and the building was of the same vintage of a brownstone. Granted, I rarely got to crawl on much because there wasn't much in the way of floors, but I was exposed to all manner of dust. I can't remember (obviously) what my parents did about the lead that had to be there, but I wasn't effected. From what I understand, lead doesn't really leave the system, and a couple of years ago I got tested a couple times and there was no noticeable amount.
If you really like your place, don't move. There are risks everywhere, you're just going to be trading in for different ones. Don't sand the wall, paint it, heck have professionals come in if you want, but don't panic. I have a brand new baby myself, and I wouldn't move for that. Oh, and don't let your baby sit there chewing paint chips, something I suspect you're unlikely to do anyway.