Ehow Engineering

Debugging Python Production Systems Like a Boss!

Introduction

When working with python web/api servers, it can be unavoidable to be in situations where you have to debug on production and remote servers. Luckily, there is a handy line of code that experienced python programmers often use to make hidden problems transparent and allow for efficient debugging by inspecting variables line by line via the debugger.

`import pdb; pdb.set_trace()`

However, this debugging method has a significant limitation. In most production systems, the web server operates through a gateway interface (e.g., WSGI), where the request is handled by background processes. This means that this handy trick no longer works as there’s no access to the debugger, and valuable insight into the problem remains hidden!

This limitation used to frustrate me until I discovered celery.contrib.rdb. It completely changed the game for me. With celery.contrib.rdb, I can easily open up debuggers on production systems that are running on background processes. In this blog, we will explore how to integrate celery.contrib.rdb with a Flask application (although it also works for any web server managing Python processes). This integration provides real-time, remote debugging capabilities that can greatly change the workflows of DevOps and development teams.

Setting the Scene

Consider a Flask API server we built for managing books:

from flask import Flask, request, jsonify

app = Flask(__name__)
books = []  # In-memory storage for the books

@app.route('/books', methods=['GET'])
def get_books():
    return jsonify(books)

@app.route('/books', methods=['POST'])
def add_book():
    data = request.get_json()
    if not data or not 'title' in data or not 'author' in data:
        return jsonify({'error': 'Invalid data provided'}), 400
    books.append({'title': data['title'], 'author': data['author']})
    return jsonify({'message': 'Book added successfully'}), 201

if __name__ == "__main__":
    app.run()

We can interact with this server using curl:

curl -X POST -H "Content-Type: application/json" -d '{"title": "To Kill a Mockingbird", "author": "Harper Lee"}' http://127.0.0.1:5000/books # Response -> {"message":"Book added successfully"}

curl -X POST -H "Content-Type: application/json" -d '{"title": "Moby Dick", "author": "Herman Melville"}' http://127.0.0.1:5000/books # Response -> {"message":"Book added successfully"}

curl -X POST -H "Content-Type: application/json" -d '{"title": "Harry Potter", "author": "J.K. Rowling"}' http://127.0.0.1:5000/books # Response -> {"message":"Book added successfully"}

curl -X GET http://127.0.0.1:5000/books # Response -> [{"author":"Harper Lee","title":"To Kill a Mockingbird"},{"author":"Herman Melville","title":"Moby Dick"},{"author":"J.K. Rowling","title":"Harry Potter"}]

Simulating an Issue

To make things interesting, let’s introduce an intentional error when adding a book titled “Twilight”:

@app.route('/books', methods=['POST'])
def add_book():
    try:
        data = request.get_json()
        if not data or not 'title' in data or not 'author' in data:
            raise ValueError("Invalid data provided")

        if data['title'] == "Twilight":
            raise RuntimeError("Big Problem!")

        books.append({'title': data['title'], 'author': data['author']})
        return jsonify({'message': 'Book added successfully'}), 201

    except ValueError as ve:
        return jsonify({'error': str(ve)}), 400

    except Exception as e:
        return jsonify({'error': 'Internal Server Error', 'message': str(e)}), 500

When we try to add the book “Twilight”, the server responds with an error:

curl -X POST -H "Content-Type: application/json" -d '{"title": "Twilight", "author": "Stephanie Meyer"}' http://127.0.0.1:5000/books # Response -> {"error":"Internal Server Error","message":"Big Problem!"}

Deploying the Remote Debugger

Now normally if we were using a WSGI with Apache or some other webserver to handle all the Python processes using a debugger would pretty much be impossible in it’s current setup, and usually one would have to revert back to debugging using a bunch of print() statements throughout the code to get some insight. Definitely not an ideal debugging situation. This is where celery.contrib.rdb shines.

@app.route('/books', methods=['POST'])
def add_book():
    try:
        data = request.get_json()
        if not data or not 'title' in data or not 'author' in data:
            raise ValueError("Invalid data provided")

        # Adding Remote Debugger Here
        from celery.contrib import rdb; rdb.set_trace()
        if data['title'] == "Twilight":
            raise RuntimeError("Big Problem!")

        books.append({'title': data['title'], 'author': data['author']})
        return jsonify({'message': 'Book added successfully'}), 201

By simply adding celery.contrib import rdb; rdb.set_trace()before our intentional error, the debugger halts the application execution, opening up a remote connection, and then allows you to connect to this debugger using telnet.

telnet localhost 6899
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
>

Real-time Inspection

Now, you’re greeted with a familiar pdb prompt, but it’s remotely connected to your server, and with the debugger active, you can inspect variables, step through the code, and identify issues, just like you would if you were using the standard pdb.

(Pdb) l
 19                 raise ValueError("Invalid data provided")
 20  
 21             # Intentionally raising an error to simulate a server issue
 22             from celery.contrib import rdb
 23             rdb.set_trace()
 24  ->         if data['title'] == "Twilight":
 25                 raise RuntimeError("Big Problem!")
 26  
 27             books.append({'title': data['title'], 'author': data['author']})
 28             return jsonify({'message': 'Book added successfully'}), 201
 29  
(Pdb) p data['title']
'Twilight'

Utilizing celery.contrib.rdb for troubleshooting, live systems offer a distinct advantage by enabling teams to swiftly pinpoint problems at the root without the need to set it up locally or in development environments after removing web server setups to allow devs to access the pdb. Complete game changer!

Conclusion

Remote debugging tools like celery.contrib.rdb are essential for today’s dynamic and distributed applications. By integrating these tools into our development and deployment processes, we ensure a rapid response to issues, minimize disruptions and maintain a high level of service quality. Just remember to remove or disable the debugger in production once the issue is resolved to prevent unintended access or performance issues.

Happy Debugging!

Email: [email protected]
Linkedin: https://www.linkedin.com/in/eric-howard-8a4166127/

domhoward14