The Federal Reserve and other analysts warn of negative impacts on growth and higher consumer prices due to the tariffs.