I had a new battery that was faulty also. Ended up shipping it back and getting a refund. Took me a while to figure out that it was the battery ("It's brand new, it can NOT be the battery") lol
Yea if you set it for volts you can measure it. Mine has several ranges for volts, always go from a large value towards smaller until you get a reading, not the other way. I set mine in the 20V setting, next options are something in the order of 300V or 1V or something.
Kicking the kicker doesn't actually pull much from the battery. But you can see what happens while you kick it or press the button anyways. I thought my starter button didn't do anything for a while either, but it still dropped the voltage over the battery when I pressed it. Anyways if your battery ever gets below 9V, it's dead. Charged, without any load, it should hold about 12.7V. Anything under like 12.4V is an empty battery. If your starter button doesn't do anything and you're still unsure about your battery, you could get it load tested at a bike shop as well.