Yeah your buddy is totally right, it can cause big problems for them. I ran into the same thing last summer and felt bad about it. What worked for me was just asking the crew lead where it was okay to take stuff from and where to leave it. They pointed me to a scrap pile they didn't need and were actually cool about it once I asked.