You are viewing a single thread.
View all comments View context
3 points
*

We looked into it a bit, and it seems it’s not freeing test case instance data (this bug, and some related bugs). We use unittest-style tests, but use pytest as a runner, and our test suite is all kinds of bloated. Basically, a typical test looks like this:

def BaseTest(unittest.TestCase):
    def setUp(self):
        for fixture in self.FIXTURES:
            db.add(fixture)

    def tearDown(self):
        db.clear()  # I'm pretty sure this actually clears memory, so not our issue

def MyTest(BaseTest):
    FIXTURES = [ ... large list of data ... ]

    def test_example(self):
        result = Service.do_operation()  # uses FIXTURES
        ... asserts ...

My best guess is that FIXTURES (and anything on the TestCase) is being kept in memory, and we do this everywhere.

Our plan is to separate our service layer from our db layer, which will dramatically reduce memory in these test case instances (we’re doing it anyway for other reasons). At the same time, we are considering porting to pytest-style tests, which will give us an opportunity to rearchitect how we handle fixtures. Some of our repos have already switched, but our larger repos haven’t.

That said, this is pretty old information. We ran into these issues around the end of last year, and haven’t looked at it since, but will in the next couple of months (we do Python upgrades around year end). It’s possible it is already fixed, either in Python or pytest (bug is still open, but no activity for a year).

Anyway, if you run into weird memory issues when running tests w/ Python >=3.12, this might be the culprit.

permalink
report
parent
reply
2 points

Thanks for sharing. We use all pytest-style tests using pytest fixtures. I’ll keep my eyes open for memory issues when we test upgrading to python 3.12+.

Very helpful info!

permalink
report
parent
reply

Python

!python@programming.dev

Create post

Welcome to the Python community on the programming.dev Lemmy instance!

📅 Events
Past

November 2023

October 2023

July 2023

August 2023

September 2023

🐍 Python project:
💓 Python Community:
✨ Python Ecosystem:
🌌 Fediverse
Communities
Projects
  • Pythörhead: a Python library for interacting with Lemmy
  • Plemmy: a Python package for accessing the Lemmy API
  • pylemmy pylemmy enables simple access to Lemmy’s API with Python
  • mastodon.py, a Python wrapper for the Mastodon API
Feeds

Community stats

  • 733

    Monthly active users

  • 469

    Posts

  • 2.4K

    Comments