Ошибка maximum recursion depth exceeded while calling a python object

I’ve built a crawler that had to run on about 5M pages (by increasing the url ID) and then parses the pages which contain the info’ I need.

after using an algorithm which run on the urls (200K) and saved the good and bad results I found that the I’m wasting a lot of time. I could see that there are a a few returning subtrahends which I can use to check the next valid url.

you can see the subtrahends quite fast (a little ex’ of the few first «good IDs») —

510000011 # +8
510000029 # +18
510000037 # +8
510000045 # +8
510000052 # +7
510000060 # +8
510000078 # +18
510000086 # +8
510000094 # +8
510000102 # +8
510000110 # etc'
510000128
510000136
510000144
510000151
510000169
510000177
510000185
510000193
510000201

after crawling about 200K urls which gave me only 14K good results I knew I was wasting my time and need to optimize it, so I run some statistics and built a function that will check the urls while increasing the id with 818178 (top returning subtrahends ) etc’.

this is the function —

def checkNextID(ID):
    global numOfRuns, curRes, lastResult
    while ID < lastResult:
        try:
            numOfRuns += 1
            if numOfRuns % 10 == 0:
                time.sleep(3) # sleep every 10 iterations
            if isValid(ID + 8):
                parseHTML(curRes)
                checkNextID(ID + 8)
                return 0
            if isValid(ID + 18):
                parseHTML(curRes)
                checkNextID(ID + 18)
                return 0
            if isValid(ID + 7):
                parseHTML(curRes)
                checkNextID(ID + 7)
                return 0
            if isValid(ID + 17):
                parseHTML(curRes)
                checkNextID(ID + 17)
                return 0
            if isValid(ID+6):
                parseHTML(curRes)
                checkNextID(ID + 6)
                return 0
            if isValid(ID + 16):
                parseHTML(curRes)
                checkNextID(ID + 16)
                return 0
            else:
                checkNextID(ID + 1)
                return 0
        except Exception, e:
            print "somethin went wrong: " + str(e)

what is basically does is -checkNextID(ID) is getting the first id I know that contain the data minus 8 so the first iteration will match the first «if isValid» clause (isValid(ID + will return True).

lastResult is a variable which saves the last known url id, so we’ll run until numOfRuns is

isValid() is a function that gets an ID + one of the subtrahends and returns True if the url contains what I need and saves a soup object of the url to a global varibale named — ‘curRes‘, it returns False if the url doesn’t contain the data I need.

parseHTML is a function that gets the soup object (curRes), parses the data I need and then saves the data to a csv, then returns True.

if isValid() returns True, we’ll call parseHTML() and then try to check the next ID+the subtrahends (by calling checkNextID(ID + subtrahends), if none of them will return what I’m looking for I’ll increase it with 1 and check again until I’ll find the next valid url.

you can see the rest of the code here

after running the code I got about 950~ good results and suddenly an exception had raised —

«somethin went wrong: maximum recursion depth exceeded while calling a
Python object»

I could see on WireShark that the scipt stuck on id — 510009541 (I started my script with 510000003), the script tried getting the url with that ID a few times before I noticed the error and stopped it.

I was really exciting to see that I got the same results but 25x-40x times faster then my old script, with fewer HTTP requests, it’s very precise, I have missed only 1 result for 1000 good results, which is find by me, it’s impossible to rum 5M times, I had my old script running for 30 hours and got 14-15K results when my new script gave me 960~ results in 5-10 minutes.

I read about stack limitations, but there must be a solution for the algorithm I’m trying to implement in Python (I can’t go back to my old «algorithm», it will never end).

Thanks!

A Recursive function in programming is a function which calls itself. These functions find applications while constructing programs for factorial, Fibonacci series, Armstrong numbers, etc. The main idea is to break larger programs into smaller, less complex problems. With recursive functions, generating sequences becomes easy. But while using recursive functions, recursionerror may occur in python. In this article, we shall be looking into one such recursionerror: maximum recursion depth exceeded while calling a Python object

What is recursionerror?

As the name suggests, Recursionerror may occur when we are dealing with recursive functions. When we run the recursion function for a large number of times, recursion error is thrown. Python has a limit on the number of times a recursive function can call itself. This is done to ensure that the function does not execute infinitely and stops after some number of iterations. To know the recursion limit in python, we use the following code:

import sys
print(sys.getrecursionlimit())

The output is:

1000

Let us look at an example of RecursionError: maximum recursion depth exceeded. We shall take an example of a factorial function.

The following code shall generate factorial for a given number.

def find_fact(n):
  if n == 0 or n == 1:
    return 1
  else :
    return (n*find_fact(n-1))  

print("Factorial is :", find_fact(5))

Here, this program shall be executed successfully and shall print the below output:

Factorial is : 120

But if we pass a larger number into the find_fact() function, it will throw RecursionError: Maximum Recursion Depth Exceeded error.

print("Factorial is :", find_fact(5000))

Output:

RecursionError: maximum recursion depth exceeded in comparison

Since the recursion function exceeded the limit of 1000 iterations, recursionerror is thrown.

The RecursionError: Maximum Recursion Depth Exceeded error may also be thrown while we are trying to create a nested list whose length exceeds the recursion limit.

Let us take the following example. We have created a function named nested() which accepts one argument – n. Depending on the value of n, the length of that nested list would be created. Let us try to pass a value n greater than the recursion limit.

def nested(n): 
    list1 = list2 = [] 
    for i in range(n): 
        list1.append([])
        list1 = list1[0] 
    return list2

nestedlist = nested(2000)
print(nestedlist)

The output will be a recursion error.

RecursionError: maximum recursion depth exceeded while getting the repr of an object

RecursionError: Maximum Recursion Depth Exceeded While Calling A Python Object

The recursionerror for Maximum Recursion Depth Exceeded While Calling A Python Object is thrown when we are trying to call a python object in Django. The error may also occur while using Flask.

When the interpreter detects that the maximum depth for recursion has reached, it throws the recursionerror. To prevent the stack from getting overflow, python raises the recursionerror.

Best practices to avoid RecursionError: Maximum Recursion Depth Exceeded while calling a Python Object

1. Using other loops instead of recursion

To prevent the error from occurring, we can simply convert the piece of code from recursion to a loop statement.

If we take the example of the factorial function, we can convert it into a non – recursive function. We do that by placing a for loop inside the recursion function. The for loop will execute for a length equal to the value of the factorial number.

def find_fact(n):
  mul = 1
  for i in range(2,n+1):
    mul = mul * i
  return mul

print("Factorial is :", find_fact(1500))

Now, it will not throw any recursion error and simply print the large factorial number.

2. Using sys.setrecursionlimit() function

Else, if we still want to use the recursion function, we can increase the recursion limit from 1000 to a higher number. For that, we have to first import the sys library. Using the sys library, we will use the sys.setrecursionlimit() function.

import sys
sys.setrecursionlimit(2000)

Now, it will not thrown the recursionerror and the program will be executed for larger amount of recursions. On executing the recursive function, it will not throw any error and print its output.

def find_fact(n):
  if n == 0 or n == 1:
    return 1
  else :
    return (n*find_fact(n-1))  

print("Factorial is :", find_fact(1500))

3. Setting boundary conditions

It is necessary to set boundary conditions to ensures that the recursive function comes to an end. In the factorial program, the condition :

'if n == 1 or n == 0 : return 1'

is the boundary condition. It is with this condition that the loop comes to an end.

4. Creating a converging recursion

While writing the recursion condition, one has to ensure that the condition does come to an end and does not continue infinitely. The recursive calls should eventually tend towards the boundary condition.

We have to ensure that we creating a converging condition for that. In the factorial program, the ‘n*fact(n-1)’ is a converging condition that converges the value from n to 1.

5. Using Memoization

We can also use memoization to reduce the computing time of already calculated values. This way, we can speed up the calculations by remembering past calculations.

When recursive calls are made, then with memoization we can store the previously calculated values instead of unnecessarily calculating them again.

That sums up the article on RecursionError: Maximum Recursion Depth Exceeded While Calling A Python Object. If you have any questions in your mind, don’t forget to let us know in the comments below.

Until next time, Keep Learning!

Sign Up for Our Newsletters

Subscribe to get notified of the latest articles. We will never spam you. Be a part of our ever-growing community.

By checking this box, you confirm that you have read and are agreeing to our terms of use regarding the storage of the data submitted through this form.