Each college at U of T has its own history, culture and traditions that you can look at and consider. I personally am at Trinity College, which I chose for its small community and traditions, but you will have a great experience no matter where you choose. As a non-resident student, you're looking more at a culture or tradition to identify with then, over residence rooms, residence fees or location of your dorm relative to the rest of the university.

Look at what sort of clubs or atmosphere is fostered by your college. Religion may play a small role - in that Trinity College is Anglican, while St. Michael's is Catholic, etc. - although there is absolutely no rule, nor is it the majority of the case for students there that you must be religious (or practice the faith of your college) to belong to any of the colleges. If you are not religious, do not worry that this will in any way affect you. I am not religious myself - but you may for example find it convenient to attend church on Sundays at St. Michael's, if you do happen to be Catholic.

Each college hosts its own events - Trinity's Bubbly in December, for example, or UC's Fireball, and you'll know which college is best for you depending on what sort of activities you want to join in on.

Victoria College gave out way more than Trinity, you get $1000 when you get 3.5 GPA. While at Trin, you only get a couple hundred bucks when you have a 3.7GPA and you get nothing if you did not get the entrance scholarship.

UC probably have more scholarships in smaller amounts than Trin and Vic, but at least you get some share of the cake. Trinity give out a lot of money but not many people can get it because it's usually lumped into one or two scholarships.

St Mikes is probably the college with the highest number of attractive people, and it's a even bigger party college than Trin. Went to a few parties there and they were awesome.
But UC by far has the biggest and best formal ball on campus, Fireball is awesome!

just don't pick st mikes...