analyses/Hiidenportti Mask R-CNN.ipynb

Hiidenportti Mask R-CNN.ipynb

{
  "cells": [
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:37:19.373550Z",
          "start_time": "2021-10-07T10:37:18.609163Z"
        },
        "trusted": false
      },
      "id": "0522421b",
      "cell_type": "code",
      "source": "%matplotlib inline\n\nfrom pathlib import Path\nfrom drone_detector.tiling import *\nimport os, sys\nimport geopandas as gpd",
      "execution_count": 1,
      "outputs": []
    },
    {
      "metadata": {},
      "id": "1b878c99",
      "cell_type": "markdown",
      "source": "# Introduction"
    },
    {
      "metadata": {},
      "id": "620ed9e7",
      "cell_type": "markdown",
      "source": "Deadwood and decaying wood, both standing and fallen, is a vital component for the biodiversity of forests, as it offers a home for several endangered species (such as fungi, mosses, insects and birds). However, when comparing European countries, Finland ranks on the bottom in the amount of both standing and fallen deadwood (m³/ha), with on average only 6 m³/ha of both deadwood types. There are, however, large differences between different forest types, as non-managed old-growth forests have several times more deadwood compared to managed forests.\n\n[Kattenborn2019](http://dx.doi.org/10.1016/j.rse.2019.03.025) propose that UAVs are in some cases an efficient alternative for traditional field sampling. For instance, if the goal is to estimate spatial coverage of a single species, utilizing plot-based or point-based field sampling data has several challenges, such as spatial inaccuracies and limited spatial coverage. Moreover, acquiring field data is labour intensive and costly. UAVs, on the other hand, provide continuous, georeferenced field data in larger quantities than field sampling. Because these data have very high spatial resolution, it is possible for experts to visually identify the target of interest. Of course, this type of ground reference data has its limitations, as it is only possible to identify objects that are visible from the data, and if the target is to estimate e.g. growing stock volume or basal area, field work is then needed. \n\nTo the best of our knowledge, there are no previous work about deadwood detection, standing or fallen, from UAV images using state-of-the-art object detection models. This work aims to produce a tool that can be used to generate deadwood tree species map automatically and accurately from captured RGB imagery. We will also address the shortcomings for deadwood mapping using only UAV imagery."
    },
    {
      "metadata": {},
      "id": "b469bc96",
      "cell_type": "markdown",
      "source": "# Materials and methods"
    },
    {
      "metadata": {},
      "id": "7e679dba",
      "cell_type": "markdown",
      "source": "## Study area"
    },
    {
      "metadata": {},
      "id": "aa4e3898",
      "cell_type": "markdown",
      "source": "The study area is located in the eastern side of Hiidenportti National Park, Eastern-Finland, and the field data contains areas both inside and outside the conserved areas (Hiidenportti National Park in the west and Teerisuo - Lososuo area in the east). The most common trees in Hiidenportti area are Scots pine and Norway spruce, with different deciduous trees (e.g. Silver birch, Downy birch and European aspen)."
    },
    {
      "metadata": {},
      "id": "1d4ab07b",
      "cell_type": "markdown",
      "source": "## Field data"
    },
    {
      "metadata": {},
      "id": "c34343f6",
      "cell_type": "markdown",
      "source": "The nine partially overlapping UAV orthoimages cover approximately 10 km² area, with spatial resolution ranging between 3.9cm and 4.4cm. These scenes cover both conserved and managed forests, as well as some logging openings. \n\nIn order to generate the ground reference data, an expert manually annotated around 5000 deadwood, both standing and fallen, from the images. As we have sample plots measured in the area, all deadwood was annotated in 100 times 100m (around 2300 times 2300 pixels) square centered around a plot center. After final visual interpretation to check which plots have sufficient image quality, this process resulted in 74 partially overlapping plots, with 4482 annotated fallen deadwood polygons and 409 standing deadwood polygons. From these 74 plots, 10 spatially distinct, non-overlapping plots were excluded from the training data and used only for testing the model performance."
    },
    {
      "metadata": {},
      "id": "6c9c94c4",
      "cell_type": "markdown",
      "source": "## Training data generation"
    },
    {
      "metadata": {},
      "id": "3f12041f",
      "cell_type": "markdown",
      "source": "Explanations come later."
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:40:27.894159Z",
          "start_time": "2021-10-07T10:40:27.890953Z"
        },
        "trusted": false
      },
      "id": "17af4393",
      "cell_type": "code",
      "source": "tile_folder = Path('../data/hiidenportti/raw/plot_patches/')\nvector_folder = Path('../data/hiidenportti/raw/vector_plots/')\n\noutpath = Path('../data/hiidenportti/processed/')\n\ntiles = os.listdir(tile_folder)\nvectors = [f for f in os.listdir(vector_folder) if f.endswith('shp')]\nassert len(tiles) == len(vectors)",
      "execution_count": 6,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:40:28.428798Z",
          "start_time": "2021-10-07T10:40:28.427030Z"
        },
        "trusted": false
      },
      "id": "4277a9a1",
      "cell_type": "code",
      "source": "import rasterio as rio",
      "execution_count": 7,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:41:13.948473Z",
          "start_time": "2021-10-07T10:40:29.324157Z"
        },
        "trusted": true
      },
      "id": "55311fda",
      "cell_type": "code",
      "source": "for t in tiles:\n    with rio.open(tile_folder/t) as f:\n        im = f.read()\n        tilesize = min(im.shape[1], im.shape[2])/4\n    if not os.path.exists(outpath/t[:-4]): os.makedirs(outpath/t[:-4])\n    shp_fname = f'{t.split(\"_\")[0]}.shp'\n    #tilesize = 500\n    tiler = Tiler(outpath=outpath/t[:-4], gridsize_x=tilesize, gridsize_y=tilesize, overlap=(0,0))\n    tiler.tile_raster(str(tile_folder/t))\n    tiler.tile_vector(vector_folder/shp_fname)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:41:51.360827Z",
          "start_time": "2021-10-07T10:41:51.358974Z"
        },
        "trusted": false
      },
      "id": "58544e88",
      "cell_type": "code",
      "source": "from shapely.geometry import box",
      "execution_count": 9,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:42:14.832975Z",
          "start_time": "2021-10-07T10:41:51.959640Z"
        },
        "trusted": false
      },
      "id": "6fe92e16",
      "cell_type": "code",
      "source": "# Fix labelling, todo fix it in COCOProcessor\n\nfor p in os.listdir(outpath):\n    files = [outpath/p/'vector_tiles'/f for f in os.listdir(outpath/p/'vector_tiles') if f.endswith('geojson')]\n    for f in files: \n        gdf = gpd.read_file(f)\n        bbox = box(*gdf.total_bounds)\n        gdf['geometry'] = gdf.geometry.buffer(0.05) # Two pixel buffer maybe?\n        gdf.rename(columns={'groundwood':'label'}, inplace=True)\n        gdf = gpd.clip(gdf, bbox, keep_geom_type=True)\n        gdf['label'] = gdf.apply(lambda row: 'Standing' if row.label == 1 else 'Fallen', axis=1)\n        gdf.to_file(f, driver='GeoJSON')",
      "execution_count": 10,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:42:35.337995Z",
          "start_time": "2021-10-07T10:42:18.489097Z"
        },
        "trusted": true
      },
      "id": "9ca62ee7",
      "cell_type": "code",
      "source": "# Convert to COCO format\n\nfrom drone_detector.coco import *\n\ndeadwood_categories = [\n        {'supercategory':'deadwood', 'id':1, 'name': 'Standing'},\n        {'supercategory':'deadwood', 'id':2, 'name': 'Fallen'},\n    ]\n\nfrom datetime import date\n\ncoco_info = {'description': 'Dataset for deadwood detection in Hiidenportti',\n             'version': 0.1,\n             'year': 2021,\n             'contributor': 'Janne Mäyrä',\n             'date_created': date.today().strftime(\"%Y/%m/%d\")\n}\n\ncoco_licenses = {}\n\nfor p in os.listdir(outpath):\n    coco_processor = COCOProcessor(outpath/p, outpath/p, coco_info=coco_info, coco_licenses=coco_licenses,\n                                  coco_categories=deadwood_categories)\n    coco_processor.shp_to_coco()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:42:41.482937Z",
          "start_time": "2021-10-07T10:42:39.400613Z"
        },
        "trusted": false
      },
      "id": "c7e63d7d",
      "cell_type": "code",
      "source": "# Combine several coco-annotation .json files into one\n\nimport json\n\nfull_coco = None\nimage_id_modifier = 0\nann_id_modifier = 0\nfor p in os.listdir(outpath):\n    with open(outpath/p/'coco.json') as f:\n        coco = json.load(f)\n    \n    # update filename\n    for i in coco['images']:\n        i['file_name'] = f\"{p}/raster_tiles/{i['file_name']}\"\n        \n    if full_coco is None: \n        full_coco = coco\n        image_id_modifier = full_coco['images'][-1]['id']\n        ann_id_modifier = full_coco['annotations'][-1]['id']\n    else:\n        for i in coco['images']:\n            i['id'] += image_id_modifier\n        for a in coco['annotations']:\n            a['image_id'] += image_id_modifier\n            a['id'] += ann_id_modifier\n        \n        full_coco['images'].extend(coco['images'])\n        full_coco['annotations'].extend(coco['annotations'])\n        image_id_modifier = full_coco['images'][-1]['id'] + 1\n        ann_id_modifier = full_coco['annotations'][-1]['id'] + 1\n        \nwith open(outpath/'full_hiidenportti_coco.json', 'w') as outfile: json.dump(full_coco, outfile)",
      "execution_count": 12,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:42:51.340287Z",
          "start_time": "2021-10-07T10:42:46.709982Z"
        },
        "trusted": false
      },
      "id": "47805280",
      "cell_type": "code",
      "source": "from icevision.all import *",
      "execution_count": 13,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:42:53.585497Z",
          "start_time": "2021-10-07T10:42:51.341468Z"
        },
        "trusted": true
      },
      "id": "a38f47dc",
      "cell_type": "code",
      "source": "parser = parsers.COCOMaskParser(outpath/'full_hiidenportti_coco.json', img_dir=outpath)\nclass_map = ClassMap(['Standing', 'Fallen'])\ntrain, valid = parser.parse()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "ExecuteTime": {
          "end_time": "2021-10-07T10:43:07.434021Z",
          "start_time": "2021-10-07T10:42:57.355174Z"
        },
        "trusted": false
      },
      "id": "00c8fe87",
      "cell_type": "code",
      "source": "ix = random.randint(0, (len(train)-9))\nshow_records(train[ix:ix+9], ncols=3, class_map=class_map, display_bbox=True, display_label=True)",
      "execution_count": 15,
      "outputs": [
        {
          "data": {