Do rocks affect the water pH? Yes. If your rock is actually limestone, the odds are good that they are the cause for the pH elevation in your aquarium water. Limestone is calcareous and is known for its ability to both harden the water and increase the pH. In fact, if someone asks me how to raise water pH, one method I suggest is to place limestone, crushed coral or any highly calcareous material in their filter.
Your water may also be naturally hard and alkaline, which adds further to the high pH. I'd definitely recommend testing the rock and if there is any doubt, remove it entirely. If the tank stabilizes, you've found the culprit.
If you don't want to give up on the rocks you have, you should determine what they're composed of. Test the composition of your rock by putting a few drops of ordinary white vinegar on it. If the vinegar foams, the rock is calcareous and this will affect the hardness and pH of the water. Limestone is probably the most frequently encountered calcareous rock, followed closely by marble.
Another means of testing an unknown rock is to fill a bucket with water from the water source you plan to use. Then test and record the pH, hardness, nitrates and phosphates. Place the rock in the bucket and let it soak for a week, then test the water again. If there are no changes, the rocks are not likely to cause a problem in your aquarium. If the parameters change considerably, I'd advise against using the rock in question.
You haven't mentioned what kind of fish you are planning to keep. Although pH can be altered, I don't advise attempting to make major changes in the pH of your water. It is very difficult to keep the pH stable over time, and you may be faced with an even more dangerous situation as a result: fluctuating pH. Changes in pH are stressful -- if not lethal -- to your fish. Furthermore, sudden changes in pH can damage the beneficial bacterial colonies that eliminate wastes in your aquarium.
In other words, keeping the pH at a steady state is just as important as the actual pH value itself. I suggest choosing fish that thrive in the pH of the water source that you have, or finding a water source that is already in the range you desire, rather than attempting to make major changes in the pH. If your water is naturally hard and alkaline, choose fish such as African cichlids that thrive in that environment. If your water is naturally soft and acidic, consider fish such as members of the popular Tetra family, almost all of which relish that type of water.
Lastly, the nitrite level you're experiencing is to be expected because you're three weeks into a newly set-up aquarium. The rocks aren't causing any increase there. Occasionally, some aquariums get stuck on the nitrite portion of the nitrogen cycle, but odds are you will see nitrites drop within the next week or two. Make sure you keep an eye on the nitrite levels because they can be lethal if they get too high. Perform regular water changes to keep them from raising to dangerous levels. The aquarium should be relatively stable when the nitrates fall, and it will remain so as long as you keep up with proper maintenance.
Good luck with your new tank. Send a photo when you get stocked.