The option that you should try when figuring out what to wear with your fedora hat is the matching loungewear set. You can elevate this look by wearing a trendy jacket, sneakers and simple jewelry.