Python GC Settings - Change This and Go 20% Faster!

Поділитися
Вставка
  • Опубліковано 29 жов 2024

КОМЕНТАРІ • 18

  • @daincrawford8439
    @daincrawford8439 Рік тому +3

    Just ran this on one of my scripts with the three lines added. 5 min .3 sec before and 4 min 17.9 sec after. That is amazing.

    • @mikeckennedy
      @mikeckennedy  Рік тому

      Dain, that is awesome. Thanks for sharing your findings.

  • @mrmagnetic927
    @mrmagnetic927 Рік тому +1

    Always a big fan of Mike and these amazing videos. I learn so much in this space

  • @christophertyler1882
    @christophertyler1882 2 роки тому +2

    I have to give this a try. A lot of my data does come from a database but I tend to put into a Pandas dataframe.

    • @mikeckennedy
      @mikeckennedy  2 роки тому

      Excellent. Please report back on whether it makes a difference for you (positive or negative). It's one of those your milage may vary things. With Pandas, the data is in C and outside the GC's purview so it might matter less.

  • @r-tierney
    @r-tierney Рік тому +2

    Nice one! I'll definitely be trying this out. One small thing I'd probably recommend using time.monotonic() to avoid time drift due to NTP / daylight savings etc. when measuring this kind of stuff.

  • @MakeDataUseful
    @MakeDataUseful 2 роки тому

    Thanks for the share Michael, I have some very long running data pipelines that I think could benefit from this.

    • @mikeckennedy
      @mikeckennedy  Рік тому

      You're welcome! Let us know how it turns out.

  • @jeanchristophe15
    @jeanchristophe15 Рік тому

    Awesome video Michael! Thank you!

  • @elev007
    @elev007 2 роки тому +1

    Thanks Michael!

  • @blazkowicz666
    @blazkowicz666 9 місяців тому

    Is this valid for python 3.11+?

  • @topcivilian
    @topcivilian Рік тому

    Why aren't the memory settings optimized by default?

  • @qozia1370
    @qozia1370 Рік тому

    thank you

  • @jamesfitzpatrick9607
    @jamesfitzpatrick9607 2 роки тому +1

    A question I need to ask is why are those numbers so low?

    • @mikeckennedy
      @mikeckennedy  2 роки тому

      What numbers are you talking about specifically? And what are you comparing them to?

    • @jamesfitzpatrick9607
      @jamesfitzpatrick9607 2 роки тому +1

      @@mikeckennedy kinda all the numbers but mainly the 700. With thinking about this a bit sense I guess this could be a bit of a historical number. On older machines with lower memory, higher numbers could cause problems. So using a 50,000 could be costly. I am now wondering if those numbers will be changed by default in the future as python is trying to make performance improvements.

    • @mikeckennedy
      @mikeckennedy  2 роки тому +1

      @@jamesfitzpatrick9607 I see. It's interesting that it's 700. That seems insanely low to me. I bet you're right that in 1991 that made sense. It doesn't now. Plus, the thing to keep in mind is that it *only* applies to cycles that are missed by ref counting. Ref counting runs first, then if there are cycles that would have leaked, then the GC kicks in. For 95% of apps, that almost never happens. I'm glad they make it configurable.