As a Canadian business owner, there are several resources and procedures you need to understand. I have compiled a list of the main ones here.
Registration
When starting your business, you need to register with your province. You will pay a small fee and receive your certification. Here is the link for registering your business if it's a sole proprietorship or partnership:
https://www.canada.ca/en/services/business/start/register-with-gov/register-sole-prop-partner.html
When your gross sales reach 10,000$, you will need to register on the Federal website and begin charging your customers taxes (you DO NOT need to charge taxes prior to reaching 10K in sales, but you can if you so decide).
Resources
The Canadian company you can use to print your thank you cards and other marketing materials (business cards, flyers, etc) is Vistaprint. They often have sales, which you can find out about by subscribing to their newsletter.