In the cities and towns for which they are elected, port wardens shall have exclusive jurisdiction in all matters pertaining to their duties, as specified in this subchapter and subchapters III, IV and V. Any other person who performs or attempts to perform any such duties in any city or town wherein there is a port warden forfeits for each offense $100, to be recovered in a civil action by any prosecutor.