BeautifulSoup is probably your best option. I'm not sure whether you'll be able to automatically parse out just the main content since the footer text is also wrapped in a similar element. If you just parse the 'p' elements it will be relatively clean but might include some descriptive things from the footer
You can test with Trafilatura Trafilatura is a Python package designed to gather text on the Web. It includes discovery, extraction and text processing components.
