We would love to move our farm to Poland in the future
You are going to find that the deck is strongly stacked against small farmers in Poland. The main thrust of agricultural policy in Poland is to drastically reduce the number of small farms and production of raw staples to meet EU goals. You will also have to deal with mountains of red tape, regulations and production quotas, both from the Polish government and from the EU. If you decide to produce high-value products for sale in one of the wealthier EU countries, which is almost certainly what you would have to do, the recipient country will be another source of red tape and regulation.
If you have inherited land already, you might be better off renting the land out or planting it with trees and collecting money from the government for doing so.
The EU is pretty harshly trying to reduce overproduction and inefficient small operations.
Importing livestock from the EU from the US is going to be a costly nightmare that you certainly don't want to deal with.
Also, neither you nor your husband have anything to offer that would be especially attractive or well paid on the highly competitive Polish job market. University jobs pay peanuts, and the best your husband could do is teach English or work in a call center, again for peanuts. Getting any sort of work at all is going to be difficult if you do not have Polish or other EU citizenship.
Sorry, but I cannot think of a single reason to abandon a successful small farm in the States to try to establish one in a hostile environment like Poland, or anywhere else in the EU, for that matter.
I also can't think of a good reason to buy a vacation house in Poland. It would almost certainly be cheaper and easier to rent.
If you are still determined to try this, you will need a) ABUNDANT capital to start the farm and support yourselves for the many years before you turn your first penny in profit on the very safe assumption that neither you nor your husband will be able to earn a substantial amount of money in Poland; b) a very good, knowledgeable and experienced agricultural consultant who can help you come up with a viable plan; and c) thorough knowledge of Polish and EU agricultural regulations.
Possibilities may exist in niche markets for products like medicinal herbs, organic fruits and vegetables, or high value products like artisan cheese, all for export to the richer EU countries, especially Germany. You would have to do abundant market research and networking to lay the groundwork, with a start date of at least three years from now.