Python Object Storage in Azure Redis Cache

Question

Python Object Storage in Azure Redis Cache

Sai 0

I have python application which is used for load testing of API (works with locust in backend) ..i would like to store all locust environment , runner and stats data which are python objects(Class) and non serializable which cannot be stored in redis in straightforward way...can you please suggest what could be the approach ? You can see what I mean by environment,runner in https://docs.locust.io/en/stable/use-as-lib.html

Manasa Akula 680 Reputation points Microsoft External Staff Moderator

2025-04-25T05:50:08.0833333+00:00
Hi Sai,

Greetings!

In addition to Sina Salam

Please note this Here are a couple of options you might consider:

As you've mentioned is about how to store non-serializable Python objects in Azure Redis Cache. I understand that you are using a Python application for load testing APIs with Locust and need to store environment, runner, and stats data, which are Python objects and not straightforwardly serializable.

To store these objects in Redis, you can use serialization techniques. Here are a few approaches you can consider:

Hey there! It sounds like you're having a bit of trouble storing your non-serializable Python objects in Azure Redis Cache while using Locust for load testing. Here are a few approaches you might consider:

Using Pickle for Serialization: This is a common method to serialize and deserialize Python objects. If your classes can't be easily transformed into a serializable format, using the **pickle** module can help. Just be aware that using Pickle comes with security considerations, particularly when loading data.

Using JSON for Serializable Parts: If your objects can be converted to JSON, you can use the json module. However, JSON is limited to serializing basic data types and may not work for complex objects.

3.Redis-Protobuf: If you're comfortable with Protocol Buffers, you might want to look into using **redis-protobuf**, which allows you to serialize your data into a more compact binary format.

4.Custom Serialization Methods: If you only need specific data from those objects (e.g., metrics, config parameters), consider extracting and storing just that data in a JSON-serializable format. This keeps things lightweight and more portable. This gives you the most control over how your objects are handled.

Consider Alternative Storage: If Redis proves too limiting for your use case, think about using different storage options like Azure Blob Storage or Azure Cosmos DB, both of which might support the storage and retrieval of your complex objects more seamlessly.

Could you let us know which of these approaches you find appealing, or if you've already tried any of them? Additionally, do you have any error messages or specific challenges you've encountered with your current method? That could help narrow down the best path forward.

I hope this information helps. Please do let us know if you have any further queries.

If the comment is helpful, please click "Upvote it".
Manasa Akula 680 Reputation points Microsoft External Staff Moderator

2025-05-02T00:49:20.22+00:00

Hi Sai,

I hope you're doing well! Just checking in to see if you had a chance to try out the example script and references I shared earlier. Let me know if you ran into any issues or if there's anything else I can help with.

Happy to support further if needed.

Best regards,

Manasa

1 answer

Your answer

Manasa Akula 680 Reputation points Microsoft External Staff Moderator

2025-05-02T00:49:20.22+00:00

Hi Sai,

I hope you're doing well! Just checking in to see if you had a chance to try out the example script and references I shared earlier. Let me know if you ran into any issues or if there's anything else I can help with.

Happy to support further if needed.

Best regards,

Manasa

Answer 1

Sina Salam 22,031 Volunteer Moderator

Hello Sai,

Storing non-serializable Python objects in Redis can be challenging, but there are a few approaches you can consider such as:

Serialization with Pickle.
Using JSON for Serializable Parts.
Use redis-protobuf to serialize and store data.
or would you like to implement custom serialization methods for your classes.

Let me know your preferences or the previous you've already using, and I will revert shortly.

Sai 0 Reputation points

2025-04-25T18:36:21.48+00:00

Thanks @Sina Salam and @Manasa Akula for suggesting approaches .I tried pickle to store python object environment but got Type Error: cannot pickle gevent.gevent.cevent. Event' object .I tried using json on serializable parts of object but loosing metrics as some depends on non serializable parts.Also used json pickle module and able to store in redis but while converting back to python object and getting its attributes I got Attribute error : attribute 'hub' of 'gevent._gevent__c_abstract_linkable_AbstractLinkable' objects is not writable .As last option in your solutions... redis protobuf need to check
Sai 0 Reputation points

2025-04-25T18:54:51.4033333+00:00

@Sina Salam and @Manasa Akula as i mentioned can you please check and let us know what could be the best approach to store python objects i.e. environment (env = Environment(user_classes=[MyTestUser])) and runner(runner = env.create_local_runner()) in redis (Need for redis only not going to another storage options) as described in https://docs.locust.io/en/stable/use-as-lib.html

Sina Salam 22,031 Volunteer Moderator

Hello Sai,

Thank you for more information and your feedback is highly appreciated.

I understand that Deserialized objects lose critical runtime context due to gevent’s event loop and partial serialization with JSON leads to data loss in metrics. Based on the constraint of using Redis only and considering gevent serialization limitations.

The best approach is to store configuration and metrics only and reconstruct objects dynamically.

Extract and store only serializable attributes of Environment and Runner to rebuild them later.
Use Locust’s event system to stream metrics directly to Redis without storing objects.
Rebuild the environment and runner using stored configurations and replay metrics.

# STEP 1 - Store Only Serializable Configuration**
import redis, json
r = redis.Redis()
config = {
    "user_classes": ["MyTestUser"],
    "host": "https://api.example.com",
    "num_users": 100,
    "spawn_rate": 10
}
r.set("locust:config", json.dumps(config))

# Step 2 - Store Metrics in Real Time via Event Hook**
from locust import events
import json
@events.request.add_listener
def on_request(request_type, name, response_time, response_length, exception, context, **kwargs):
    metric = {
        "type": request_type,
        "name": name,
        "response_time": response_time,
        "response_length": response_length,
        "exception": str(exception) if exception else None
    }
    r.rpush("locust:metrics", json.dumps(metric))

# Step 3 - Reconstruct the Environment from Redis
from locust.env import Environment
from my_locust_users import MyTestUser  # Ensure available in global scope
config = json.loads(r.get("locust:config"))
env = Environment(user_classes=[globals()[cls] for cls in config["user_classes"]],
                  host=config["host"])
runner = env.create_local_runner()

# Step 4 - Analyze Metrics Later
metrics = [json.loads(m) for m in r.lrange("locust:metrics", 0, -1)]
# Now you can analyze, store, or report on the metrics

For more reading:

Locust Environment Setup -https://docs.locust.io/en/stable/use-as-lib.html)
Locust Event Hooks - https://docs.locust.io/en/stable/api.html#events
Locust Programmatic Usage - https://docs.locust.io/en/stable/use-as-lib.html

NOTE: Due to Redis-Only any other attempt for partial object serialization (not recommended), will likely fail for objects with gevent internals.

Success.

Sai 0 Reputation points

2025-04-27T20:46:27.4966667+00:00

Thanks a lot @Sina Salam for your solution and example script.This should work.Will go through the references shared with example script to workout in my case.In case, if still getting issues will let you know in 1-2 days.
Manasa Akula 680 Reputation points Microsoft External Staff Moderator

2025-04-28T01:47:15.1233333+00:00
Hi Sai,

Sorry for the inconvenience caused while trying pickle to store python object.

Thank you for providing detailed feedback about the challenges you encountered while trying to store Locust Environment and Runner objects in Azure Redis Cache. I understand that you’ve faced issues with pickle (TypeError due to non-picklable gevent objects), json (loss of metrics due to non-serializable parts), and jsonpickle (AttributeError when reconstructing objects), and now considering redis-protobuf while sticking exclusively with Redis.

The core issue lies in the fact that Locust’s Environment and Runner objects contain complex, non-serializable components (e.g., gevent event loops and abstract linkable objects) that are tightly coupled to the runtime state and cannot be easily serialized or deserialized without losing functionality. After reviewing the Locust documentation (https://docs.locust.io/en/stable/use-as-lib.html) and the errors you’ve shared, I recommend the following approach to effectively store these objects in Redis:
Recommended Approach: Custom Serialization with Selective Data Extraction

Since direct serialization of Environment and Runner objects is problematic due to their reliance on gevent and other non-serializable components, the most reliable solution is to extract the relevant, serializable data from these objects and store it in Redis using a JSON-compatible format. This approach avoids the issues with pickle, jsonpickle, and gevent-related errors while preserving the metrics and configuration data you need.
Steps to Implement:

Identify Relevant Data to Store:

For the Environment object (env = Environment(user_classes=[MyTestUser])):

Configuration data: env.host, env.user_classes, env.tags, env.exclude_tags, env.shape_class, etc.

State data: Any custom attributes or settings you’ve added to the environment.

For the Runner object (runner = env.create_local_runner()):

Metrics: runner.stats (contains request statistics, failures, etc.), runner.user_count, runner.state.

Configuration: runner.target_user_count, runner.spawn_rate, etc.

The stats object (runner.stats) is particularly important, as it contains serializable data like total (aggregate stats), entries (per-request stats), and failures.

2.Extract and Serialize Data:

Write a custom serialization function to extract the relevant attributes from Environment and Runner into a dictionary.

Use the json module to serialize this dictionary into a JSON string, which is fully compatible with Redis.

Example code:

import json from locust import Environment, HttpUser class MyTestUser(HttpUser): # Your user class definition pass # Create Locust environment and runner env = Environment(user_classes=[MyTestUser]) runner = env.create_local_runner() def serialize_locust_data(env, runner): env_data = { "host": env.host, "user_classes": [cls.__name__ for cls in env.user_classes], "tags": env.tags, "exclude_tags": env.exclude_tags, # Add other relevant attributes } runner_data = { "user_count": runner.user_count, "state": runner.state, "stats": { "total": runner.stats.total.get_stripped_report(), # Aggregate stats "entries": {k: v.to_dict() for k, v in runner.stats.entries.items()}, # Per-request stats "failures": {k: v.to_dict() for k, v in runner.stats.failures.items()}, # Failure stats }, # Add other relevant attributes } return { "environment": env_data, "runner": runner_data } # Serialize to JSON serialized_data = json.dumps(serialize_locust_data(env, runner)) # Store in Redis import redis redis_client = redis.Redis(host='your_redis_host', port=6379, decode_responses=True) redis_client.set("locust_data", serialized_data)

3.Deserialize and Reconstruct:

When retrieving data from Redis, deserialize the JSON string back into a Python dictionary.

Reconstruct the Environment and Runner objects by reinitializing them and populating the relevant attributes. Note that some runtime-specific attributes (e.g., gevent event loops) cannot be restored, so focus on restoring configuration and metrics.

Example code:

# Retrieve from Redis serialized_data = redis_client.get("locust_data") deserialized_data = json.loads(serialized_data) # Reconstruct Environment env = Environment(user_classes=[MyTestUser]) # Reinitialize with same user classes env.host = deserialized_data["environment"]["host"] env.tags = deserialized_data["environment"]["tags"] env.exclude_tags = deserialized_data["environment"]["exclude_tags"] # Reconstruct Runner (limited to metrics and configuration) runner = env.create_local_runner() runner.user_count = deserialized_data["runner"]["user_count"] # Restore stats manually if needed (stats are typically read-only, so use for reporting)

4.Store in Redis:

Use Redis strings (SET/GET) to store the JSON-serialized data,or use Redis hashes (HSET/HGET) if you want to store individual fields for easier access.

Example with Redis hashes:

redis_client.hset("locust_data", "environment", json.dumps(serialized_data["environment"])) redis_client.hset("locust_data", "runner", json.dumps(serialized_data["runner"]))

Why This Works:

Avoids Non-Serializable Issues: By extracting only serializable data (e.g., configuration, metrics), you bypass the gevent-related errors encountered with pickle and jsonpickle.

Preserves Metrics: The runner.stats object provides detailed metrics in a serializable format (via get_stripped_report() or to_dict()), ensuring you don’t lose critical load-testing data.

Redis-Compatible: JSON is lightweight and natively supported by Redis, making storage and retrieval straightforward.

Flexible: You can extend the serialize_locust_data function to include additional attributes as needed.

Why Other Approaches Are Less Suitable:

Pickle: As you experienced, pickle fails due to gevent objects (e.g., cevent.Event) that are not picklable. Even if pickling were possible, deserializing and restoring the runtime state of Environment and Runner would be unreliable due to gevent’s event loop dependencies.

JSON (Direct): Direct JSON serialization loses non-serializable parts, as you noted, leading to incomplete metrics.

Jsonpickle: While jsonpickle can serialize complex objects, the AttributeError you encountered ('hub' is not writable) indicates that gevent’s internal state cannot be properly reconstructed, making this approach unreliable.

Redis-Protobuf: While redis-protobuf offers compact binary serialization, it requires defining Protocol Buffer schemas for your objects, which can be complex for Locust’s dynamic and gevent-dependent classes. Additionally, it doesn’t solve the core issue of non-serializable gevent components, and the setup overhead may outweigh the benefits for your use case.

Additional Considerations:

Data Granularity: If you only need specific metrics or configuration (e.g., runner.stats.total.get_stripped_report()), store only those to minimize data size and complexity.

Redis Key Structure: Use a structured key naming convention (e.g., locust:test_id:environment, locust:test_id:runner) to organize data for multiple tests.

Performance: JSON serialization/deserialization is lightweight, but if you’re storing large volumes of stats data, consider compressing the JSON string (e.g., using zlib) before storing it in Redis.

Security: Ensure your Redis instance is secured (e.g., with TLS and authentication) to protect sensitive load-testing data.

Next Steps:

Try the Custom Serialization Approach: Implement the example code provided above and test storing/retrieving data in Redis. Let us know if you encounter any specific errors or need help adapting the code to your setup.

Share More Details: If you have specific metrics or attributes you need to preserve, or if you’re using custom extensions with Locust, please share those details so we can refine the serialization function.

Redis-Protobuf Exploration: If you’d like to explore redis-protobuf, I can provide guidance on setting up Protocol Buffers, but I recommend trying the JSON-based approach first due to its simplicity and compatibility.

I hope this approach addresses your needs! Please let me know if you have further questions, encounter any issues while implementing this, or need additional clarification, your feedback is highly appreciated.
Sai 0 Reputation points

2025-04-28T20:55:31.3733333+00:00

No problem and Thanks a lot @Manasa Akula for your comprehensive solution. I will workout and let u know. Also in my context stats refer to env.stats
Manasa Akula 680 Reputation points Microsoft External Staff Moderator

2025-04-29T00:54:30.9133333+00:00

Hi Sai,

You're very welcome! I'm glad I could help. Thanks for clarifying that you're referring to env.stats that makes sense. Feel free to reach out if you run into any issues while implementing or need help extracting specific parts of the stats data. I’ll be happy to assist further!

Looking forward to hearing how it goes. And I Hope it will work for you.
If the comment is helpful, please click "Upvote it".
Manasa Akula 680 Reputation points Microsoft External Staff Moderator

2025-05-05T02:36:19.2066667+00:00

Hi Sai,

I hope you're doing well! Just checking in to see if you had a chance to try out the example script and references I shared earlier. Let me know if you ran into any issues or if there's anything else I can help with.

Happy to support further if needed.

Best regards,

Manasa

Share via

Python Object Storage in Azure Redis Cache

1 answer

Your answer