Update

README.md

@@ -1,3 +1,84 @@
-# ICL-IoT-Weather
+# Imperial College London
+# Sensing and IoT Coursework Project
+#### Reducing water waste by automatically dispensing lower bound water amounts, calculated using a ML Model using present weather data
 
-# TODO
+## About
+This project aims to integrate hardware IoT sensors (some RPi Zero's with I2C probes) and Machine Learning (Deep-NN) to predict how much water a outdoor garden will need every day.
+
+The ML Model uses the last 24hrs of collected weather data to make a prediction around mow much water needs to be dispensed to keep the plants alive.
+
+This approach hopes to reduce the overall water use by not watering plants in excess.
+
+## Project diagram
+```TODO```
+
+## File Structure
+
+<details>
+<summary>This part is long, click to expand</summary>
+
+- `data_collector/` Main data collector and uploader module
+  - `secrets/` Private API keys
+    - `icl-iot-weather-firebase-adminsdk.json` Firebase key for database access
+    - `weather_api_key.txt` OpenWeather API key
+  - `data_colelctor.py` Python script for data collection
+  - `Dockerfile` Containerizing the application
+  - `requirements.txt` Python requirements for running the script
+- `data_processor/` ML on demand data processing module
+  - `firebase_python_image/` Since building a tf image takes ages, we build it once and use it for testing the ml script later
+    - `Dockerfile` Building a tensorflow image for armv7l
+  - `secrets/` Private API keys
+    - `icl-iot-weather-firebase-adminsdk.json` Firebase key for database access
+  - `watering_model.model/` Saved ML modelfor water predicitons
+  - `Dockerfile` Containerizing the application
+  - `water_predictor.py` On demand, real time watering predictor script
+- `Diagrams/` Process and block diagrams
+  - `block_and_process_diagrams.drawio` TODO
+- `lora_nodes/` Scripts running on the Pi Zero nodes
+  - `master/` Scripts running on the primary, internet connected Pi
+    - `irrigator/` Daily watering module
+      - `Dockerfile` Containerizing the application
+      - `irrigator.py` Daily watering script
+      - `requirements.txt` Python requirements for running the script
+    - `Dockerfile` Containerizing the application
+    - `main.py` Main data server, collects data from satellite over LoRa and returns over http
+    - `requirements.txt` Python requirements for running the script
+  - `satellite/` Data reading and sending module for the sensor Pi
+    - `Dockerfile` Containerizing the application
+    - `main.py` LoRa commend listener, temperature reader and pump controller
+    - `requirements.txt` Python requirements for running the script
+- `manual_data_processing/` iPython notebooks used for data processing and model training
+  - `datasets/*` Various datasets used for processing and training
+  - TODO
+- `site/*` Monitoring website, HTML+CSS+JS, hosted on Firebase
+- `.gitignore` Gitignore file preventing all my API keys from showing up online...
+- `README.md` See `README.md`
+
+</details>
+
+## Installation and running
+1. Clone this repo
+2. Create the secrets folders
+3. Generate a set of firebase and OpenWeather API credentials and place them in the secrets folders, rename accordingly
+4. Move the `master` folder to the main Pi
+5. Move the `satellite` folder to the sensor Pi
+6. Install Docker on all Devices
+7. Build the docker images on the relevant architectures (for me everythong was build on armv7)
+   - `data_colelctor`: `docker build -t siot-weather-collector .`
+   - `data_processor`: `docker build -t siot-data-processor .`
+   - `master/irrigator`: `docker build -t master-node .`
+   - `master`: `docker build -t irrigator .`
+   - `satellite`: `docker build -t satellite .`
+8. *NOTE: The ML Dockerfile is very long and complicated because tensorflow does not play well with a 32 bit arm architeture, if you are building for x86 or arm64, you may need to change the file*
+9.  Run the docker containers:
+   - The `data_collector` on the cloud device: `docker run -d --restart always --name siot_weather_collector siot-weather-collector`
+   - The `data_processor` on the cloud device: `docker run -dp 3535:3535 --restart always --name siot_watering_predictor  siot-data-processor`
+   - The `master` on the master Pi: `docker run -dp 3333:3333 --privileged --restart always master-node`
+   - The `master/irrigator` on the master Pi: `docker run -d --privileged --restart always irrigator`
+   - The `satellite` on the sensor Pi: `docker run -d --privileged --restart always satellite`
+10. While you are free to use my ML model included in this repo, I suggest you explore the `manual_data_processing` folder and create your own.
+11. You will also need at least 24hrs of data before the model can make predictions, so the `siot-weather-collector` image must be started at least 24hrs before the others
+
+## Maintainers and Contributors
+
+   Max Hunt: [me@maxhunt.design](mailto:me@maxhunt.design)

data_collector/Dockerfile

@@ -4,4 +4,4 @@ COPY . .
 RUN pip install -r requirements.txt
 EXPOSE 80
 EXPOSE 443
-CMD "/code/Data_collector.py"
+CMD "/code/data_collector.py"

diagrams/e2e diagram.drawio

@@ -0,0 +1 @@
+<mxfile host="Electron" modified="2021-01-09T01:51:41.772Z" agent="5.0 (Macintosh; Intel Mac OS X 11_1_0) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/14.1.8 Chrome/87.0.4280.88 Electron/11.1.1 Safari/537.36" etag="gbp0hDtj5hE5ZAe5tMTm" version="14.1.8" type="device"><diagram id="QUbS_Z7dZOKaG96ER5AO" name="Page-1">7ZjbctowEIafhstkfMIhl4GQ0E4zTUtnuBZoa6uxJY+05tCnr4RlDJiC22nBdHqF/K800n76tdjq+IN0+SxJFr8ICknHc+iy4z92PM91nVD/GGVVKL3grhAiyWghOZUwZt/BjizVnFFQViskFCJBlu2KM8E5zHBHI1KKxW63ryKhO0JGIqgJ4xlJ6uqEUYxtFt5dpY+ARbGdudvzi0BKyr42ERUTKhZbkj/s+AMpBBatdDmAxLDbxfL0k+hmXRI4Nhmw4CIMsuebKet+ehu+fV6+5083drFzkuQ2X7tYXJUAgGoe9lFIjEUkOEmGldqXIucUzDSOfqr6fBAi06KrxW+AuLKbS3IUWooxTWy0norNTolczuDI+ktHEBkBHuln0zK5bE1gQT2DSAHlSneQkBBk8929J9ZC0aZfhVk3LOlfoO7WqH+BNOt4/VGeMsrQrEQBV0LWdqNibcAtYoYwzsia0UIfwIZc5yARlkdJlNHSvvb4uqF9XlSHwQ2sFm8dhFL74/C6V25Zr6Flg1ZZ1jtAPUz0+vtT3YhMY2wd67yyTUyWwVLRk28GPBIk6wQIhcs73euddnrvnEa/u3KjBw2NHrbK6MFpo78QhfBbRk/I6uI+9xv4/P6cPr+/cp+HDX3ea5XPy/fr/9jPi/3QH+m/iN11W8U9bFLWGTeTeI6p1wNNZq/Ib5f0euEfCYWMR2bxnJoP3qLoZ1LMQKl15MKVP7hv8C4fnrP0907vitmLKVHQbBOemISit2OaCoWEi3MP97n73Tp375zc3foXaA38BKYPWdYMu2ICb1OyjHOOtxQUi3j7qJc3M5ejHlxXqXf9hrW+XW/w7qFLrT1zf8yAT4BgvC7xD6/vmvm8GlJV9zlrw9er7++53fl79zT6sbq2XMe27n794Q8=</diagram></mxfile>

dnup_virtual_sensor/virtual_soil_sensor.py

@@ -1,3 +1,4 @@
+#!/usr/local/bin/python
 import logging
 
 import firebase_admin
@@ -9,15 +10,17 @@ from tensorflow import keras
 class Firebase:
     def __init__(self):
         self.creds = credentials.Certificate(
-            'icl-iot-weather-firebase-adminsdk.json')
+            'secrets/icl-iot-weather-firebase-adminsdk.json')
         firebase_admin.initialize_app(self.creds)
         self.db = firestore.client()
+        logging.debug('Initialized firebase instance')
 
     def pull_from_db(self, orderby=u'timestamp'):
         doc_ref = self.db.collection('weather_data')
         query = doc_ref.order_by(orderby,
                                  direction=firestore.Query.DESCENDING).limit(1)
         doc = query.stream()
+        logging.debug('Got doc file from firestore')
         return doc
 
     def convert_to_float(self, data):
@@ -37,11 +40,14 @@ class VirtualProbe:
     def __init__(self):
         self.firebase = Firebase()
         self.model = keras.models.load_model('virtual_probe.model')
+        logging.debug('Loaded ML model')
 
     def predict_soil_temp(self):
         feature = self.firebase.get_feature()
-        predicted_temp = self.model.predict([feature])
-        return predicted_temp
+        predicted_temp = float(self.model.predict([feature])[0][0])
+        predicted_temp_2dp = float("{:.1f}".format(float(predicted_temp)))
+        logging.debug(f'Predicted temp: {predicted_temp_2dp}')
+        return predicted_temp_2dp
 
 
 if __name__ == "__main__":

lora_nodes/master/irrigator/Dockerfile

@@ -3,5 +3,4 @@ WORKDIR /code
 COPY . .
 RUN pip install requests
 RUN pip install -r requirements.txt
-EXPOSE 3333
-CMD "/code/main.py"
+CMD "/code/irrigator.py"