![]() Any other dashboarding tool other than Streamlit could be used for visual monitoring. Redis could be replaced with any other storage backend, or even a dedicated model store like MLflow. You may also want to use a feature store such as Feast if that floats your boat. Flink, Pinot, or Bytewax could stand in for Materialize. Here are a bunch of thoughts loosely organised to give some food for thought.Įach technology has been picked for a particular purpose, but each one could be replaced with an alternative. There are a lot of rabbit holes to explore. It's also quite narrow, as it's a supervised learning task, which is arguably simpler to reason about than, say, a recommender system. This demo is only one way of doing online machine learning. # Follow the logs for a particular serviceĭocker exec -it redpanda rpk topic consume predictions -brokers=localhost:9092ĭocker compose down -rmi all -v -remove-orphans Going further ![]() Before this all happens, the service also uploads some models to Redis, which acts as a model store. This is all managed by the simulation service, which loops over the data at 10 times the actual speed the events occur in the dataset. Another event is sent once the taxi arrives. For each trip, it first emits an event indicating a taxi has departed. The first service is in charge of simulating the data stream. ✍️ The whole thing is made up of ~220 lines of Python, ~120 lines of SQL, and ~100 lines of YAML/Dockerfile. □ The system is Dockerized, which reduces the burden of connecting the different parts with each other. □□ Streamlit is used to monitor the overall system in real-time. Redis thus acts as a primitive model store. □ The inference and learning services coordinate with one another thanks to a Redis instance where the models are stored. This labelled sample is fed into each River model. Each time a taxi arrives, Materialize joins the original features with the trip duration. □ The River models are also listening to Materialize for taxi drop-offs. The predictions is sent to Redpanda, and then get picked up by Materialize. Each one makes a prediction for every taxi that departs. □ Several River models are listening to Materialize for taxi pick-ups. It keeps track of the system as a whole, builds aggregate features in real-time, and monitors the model's predictive performance. □ Materialize consumes the pick-up and drop-off topics from Redpanda, and does stream processing on top. The duration of the trip is also sent to Redpanda once the taxi arrives at its destination. An event is sent to Redpanda each time a taxi departs. □ Taxi trips are simulated with a Python script. □ Redpanda acts as a message bus, storing the data produced by the entire system.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |