Let's imagine you decide to go out on Friday night with several of your friends. You don't smoke marijuana - or do any type of drugs - but your friends do. They're still your friends, and you just let it go. You're heading to a party, and they decide to fire up joint. Everyone is having a good time - even you - when all of the sudden you see the red flashing lights behind you. You know how this ends; the officer smells marijuana, and ends up finding a baggie in the car. You correctly tell the officer that you weren't doing anything - he hasn't heard that before, right? - but you still get arrested.
So what happens from there? Possession is defined by law as "care, custody or control". The law also says that more than one person can "possess" something, which is why everyone in the car got arested. When the item is not on you, the focus is on knowledge and access. While it's not enough to show you knew there was marijuana in the car, you probably still had access to it - especially if it was out in the open. So how would you defend against that?
There are a number of ways to do that, but none of them are foolproof. Even if your friend(s) claim it was theirs, the police - and the court - may believe they are just covering for you.
The moral of the story is this: Don't think that just because you're not using drugs - whatever it is - or they aren't on you, that you are in the clear. So don't get yourself in that position. If you are able to eventually get the charges dismissed, you still had to spend money on bond, got a free trip to the jail, and had to pay a lawyer. It's a pretty high price to pay for something that you can avoid.