Semantic segmentation

remote_sensing_processor.segmentation.generate_map(x, y, reference, model, output, batch_size=32, num_workers=0, nodata=None)

Create map using pre-trained model.

Parameters:

• x (path as a string or xarray.DataArray) – Training data (x) generated by generate_tiles() function that will be used for prediction.

• y (path as a string or xarray.DataArray) – Target variable data (y) generated by generate_tiles() function that was used to train the model.

• reference (path as a string) – Raster that will be used as a reference raster to get size, transform and crs for a map. Use one of the rasters that were used for tile generation.

• model (torch.nn model or SklearnModel or path to a model file) – Pre-trained model to predict target values. You can pass the model object returned by train() function or file (*.ckpt or *.joblib) where model is stored.

• output (path as a string) – Path where to write output map.

• batch_size (int (default = 32)) – Number of samples used in one iteration.

• num_workers (int or 'auto' (default = 0)) – Number of parallel workers that will load the data. Set ‘auto’ to let RSP choose the optimal number of workers, set 0 to disable multiprocessing. Can increase training speed, but can also cause errors (e.g. pickling errors).

• nodata (int or float (optional)) – Nodata value. If not defined then nodata value of y dataset will be used.

Examples

>>> x_out, y_out = rsp.segmentation.generate_tiles(
...     x,
...     y,
...     tile_size=256,
...     shuffle=True,
...     split=[3, 1, 1],
...     split_names=['train', 'val', 'test']
...     )
>>> train_ds = [x_out, y_out[0], 'train']
>>> val_ds = [x_out, y_out[0], 'val']
>>> model = rsp.segmentation.train(
... train_ds,
... val_ds,
... model='UperNet',
... backbone='ConvNeXTV2',
... model_file='/home/rsp_test/model/upernet.ckpt',
... epochs=10,
... batch_size=32
... )
...
>>> reference = '/home/rsp_test/mosaics/landcover.tif'
>>> output_map = '/home/rsp_test/prediction.tif'
>>> rsp.segmentation.generate_map(x_out, y_out[0], reference, model, output_map)
Predicting: 100% #################### 372/372 [32:16, 1.6s/it]

>>> x_file = '/home/rsp_test/model/x.zarr'
>>> y_file = '/home/rsp_test/model/y.zarr'
>>> model = '/home/rsp_test/model/upernet.ckpt'
>>> reference = '/home/rsp_test/mosaics/landcover.tif'
>>> output_map = '/home/rsp_test/prediction.tif'
>>> rsp.segmentation.generate_map(x_file, y_file, reference, model, output_map)
Predicting: 100% #################### 372/372 [32:16, 1.6s/it]

>>> # Train model on data from Montana
>>> x_montana_files = glob('/home/rsp_test/mosaics/landsat_montana/*')
>>> y_montana_files = '/home/rsp_test/mosaics/landcover_montana/landcover.tif'
>>> x_montana, y_montana = rsp.segmentation.generate_tiles(
...     x_montana_files,
...     y_montana_files,
...     tile_size=256,
...     shuffle=True,
...     split=[3, 1, 1],
...     split_names=['train', 'val', 'test']
...     )
>>> train_ds = [x_montana, y_montana[0], 'train']
>>> val_ds = [x_montana, y_montana[0], 'val']
>>> model_montana = rsp.segmentation.train(
...     train_ds,
...     val_ds,
...     model='UperNet',
...     backbone='ConvNeXTV2',
...     model_file='/home/rsp_test/model/upernet.ckpt',
...     epochs=10,
...     batch_size=32
...     )
...
>>> # Use model to map landcover of Idaho
>>> x_idaho_files = glob('/home/rsp_test/mosaics/landsat_idaho/*')
>>> x_idaho, _ = rsp.segmentation.generate_tiles(x_idaho_files, None, tile_size=256)
>>> reference = x_idaho_files[0]
>>> output_map = '/home/rsp_test/prediction_idaho.tif'
>>> rsp.segmentation.generate_map(x_idaho, y_montana, reference, model_montana, output_map)
Predicting: 100% #################### 372/372 [32:16, 1.6s/it]

remote_sensing_processor.segmentation.generate_tiles(x, y, tile_size=128, classification=True, shuffle=False, split=[1], split_names=['train'], x_output=None, y_output=None, x_dtype=None, y_dtype=None, x_nodata=None, y_nodata=None)

Cut rasters into tiles.

Parameters:

• x (list of paths as strings) – Rasters to use as training data.

• y (path as a string or list of paths as strings) – Raster or multiple rasters to use as target variable. Can be set to None if target value is not needed.

• tile_size (int (default = 128)) – Size of tiles to generate (tile_size x tile_size).

• classification (bool (default = True)) – If True then tiles will be prepared for classification (e.g. semantic segmentation) task, else will be prepared for regression task.

• shuffle (bool (default = False)) – Is random shuffling of samples needed.

• split (list of ints or floats (optional)) – Splitting data in subsets. Is a list of integers defining proportions of every subset. [3, 1, 1] will generate 3 subsets in proportion 3 to 1 to 1.

• split_names (list of strings) – Names of split subsets.

• x_output (path as a string (optional)) – Path to save generated output x data. Data is saved in .zarr format.

• y_output (path as a string or list of paths as strings (optional)) – Path or list of paths to save generated output y data. Data is saved in .zarr format.

• x_dtype (dtype definition as a string (optional)) – If you run out of memory, you can try to convert your data to less memory consuming format.

• y_dtype (dtype definition as a string (optional)) – If you run out of memory, you can try to convert your data to less memory consuming format.

• x_nodata (int or float (optional)) – You can define which value in x raster corresponds to nodata and areas that contain nodata in x raster will be ignored while training and testing. Tiles that contain only nodata in both x and y will be omited. If not defined then nodata of first x file will be used.

• y_nodata (int or float (optional)) – You can define which value in y raster corresponds to nodata and areas that contain nodata in y raster will be ignored while training and testing. Tiles that contain only nodata in both x and y will be omited. If not defined then nodata of y file will be used.

Returns:

xarray.Dataarray

Array with generated x data.

xarray.Dataarray or list of xarray.Dataarray or None

List of arrays with generated y data - one array for each y raster.

Return type:

tuple

Examples

>>> x = ['/home/rsp_test/mosaics/sentinel/B1.tif',
... '/home/rsp_test/mosaics/sentinel/B2.tif',
... '/home/rsp_test/mosaics/sentinel/B3.tif',
... '/home/rsp_test/mosaics/sentinel/B4.tif',
... '/home/rsp_test/mosaics/sentinel/B5.tif',
... '/home/rsp_test/mosaics/sentinel/B6.tif',
... '/home/rsp_test/mosaics/sentinel/B7.tif',
... '/home/rsp_test/mosaics/sentinel/B8.tif',
... '/home/rsp_test/mosaics/sentinel/B8A.tif',
... '/home/rsp_test/mosaics/sentinel/B9.tif',
... '/home/rsp_test/mosaics/sentinel/B11.tif',
... '/home/rsp_test/mosaics/sentinel/B12.tif']
>>> y = ['/home/rsp_test/mosaics/landcover.tif',
... '/home/rsp_test/mosaics/forest_types.tif']
>>> x_file = '/home/rsp_test/model/x.zarr'
>>> y_files = ['/home/rsp_test/model/y_landcover.zarr',
... '/home/rsp_test/model/y_forest_types.zarr']
>>> x_out, y_out = rsp.segmentation.generate_tiles(
...     x,
...     y,
...     tile_size=256,
...     shuffle=True,
...     split=[3, 1, 1],
...     split_names=['train', 'val', 'test'],
...     x_output=x_file,
...     y_output=y_files,
...     x_nodata=0,
...     y_nodata=0
... )
>>> print(x_out.shape)
(12, 8704, 6912)
>>> y_landcover = y_out[0]
>>> print(y_landcover.shape)
(8704, 6912)
>>> y_forest_types = y_out[1]
>>> print(y_forest_types.shape)
(8704, 6912)

remote_sensing_processor.segmentation.test(test_datasets, model, batch_size=32, num_workers=0)

Tests segmentation model.

Parameters:

• test_datasets (list or list of lists) – Test data generated by generate_tiles() function. Each dataset is a list of 3 elements: training data (x): file path or xarray.DataArray, target variable (y): file path or xarray.DataArray, split_names: string or list of strings or ‘all’ if you need to use the whole dataset. You can provide a list of datasets to test model on multiple datasets.

• model (torch.nn model or SklearnModel or path to a model file) – Model to test. You can pass the model object returned by train() function or file (*.ckpt or *.joblib) where model is stored.

• batch_size (int (default = 32)) – Number of samples used in one iteration.

• num_workers (int or 'auto' (default = 0)) – Number of parallel workers that will load the data. Set ‘auto’ to let RSP choose the optimal number of workers, set 0 to disable multiprocessing. Can increase training speed, but can also cause errors (e.g. pickling errors).

Examples

>>> x_out, y_out = rsp.segmentation.generate_tiles(
...     x,
...     y,
...     tile_size=256,
...     shuffle=True,
...     split=[3, 1, 1],
...     split_names=['train', 'val', 'test']
...     )
>>> train_ds = [x_out, y_out[0], 'train']
>>> val_ds = [x_out, y_out[0], 'val']
>>> test_ds = [x_out, y_out[0], 'test']
>>> model = rsp.segmentation.train(
...     train_ds,
...     val_ds,
...     model='UperNet',
...     backbone='ConvNeXTV2',
...     model_file='/home/rsp_test/model/upernet.ckpt',
...     epochs=10,
...     batch_size=32
)
...
>>> rsp.segmentation.test(test_ds, model=model, batch_size=32)
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Test metric        ┃       DataLoader 0        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│      test_acc_epoch       │    0.8231202960014343     │
│     test_auroc_epoch      │    0.7588028311729431     │
│      test_iou_epoch       │    0.69323649406433105    │
│      test_loss_epoch      │    0.40799811482429504    │
│   test_precision_epoch    │    0.8231202960014343     │
│     test_recall_epoch     │    0.8231202960014343     │
└───────────────────────────┴───────────────────────────┘

remote_sensing_processor.segmentation.train(train_datasets, val_datasets, model_file, model, backbone=None, checkpoint=None, weights=None, epochs=5, batch_size=32, repeat=1, augment=False, less_metrics=False, lr=0.001, num_workers=0, classification=None, num_classes=None, y_nodata=None, **kwargs)

Trains segmentation model.

Parameters:

• train_datasets (list or list of lists) – Training data generated by generate_tiles() function. Each dataset is a list of 3 elements: training data (x): file path or xarray.DataArray, target variable (y): file path or xarray.DataArray, split_names: string or list of strings or ‘all’ if you need to use the whole dataset. You can provide a list of datasets to train model on multiple datasets.

• val_datasets (list or list of lists or None) – Validation data generated by generate_tiles() function. Each dataset is a list of 3 elements: training data (x): file path or xarray.DataArray, target variable (y): file path or xarray.DataArray, split_names: string or list of strings or ‘all’ if you need to use the whole dataset. You can provide a list of datasets to validate model on multiple datasets. Can be set to None if no validation needed.

• model_file (path as a string) – Checkpoint file where model will be saved after training. File extension must be *.ckpt for neural networks and *.joblib for scikit-learn models.

• model (str) – Name of model architecture.

• backbone (str (optional)) – Backbone, solver or kernel of a model, if multiple backbones are supported.

• checkpoint (path as a string (optional)) – Checkpoint file (*.ckpt or *.joblib) of a pre-trained model to fine-tune.

• weights (str) – Name of pre-trained weights to fine-tune. Only works for neural networks.

• epochs (int (default = 5)) – Number of training epochs. Only works for neural networks and multilayer perceptron.

• batch_size (int (default = 32)) – Number of training samples used in one iteration. Only works for neural networks.

• repeat (int (default = 1)) – Increase size of a dataset by repeating it n times.

• augment (bool (default = False)) – Apply augmentations to dataset.

• less_metrics (bool (default = False)) – Sometimes Torchmetrics can freeze while calculating precision, recall and IOU. If it happens, try restarting with less_metrics = True.

• lr (float (default = 1e-3)) – Learning rate of a model. Lower value results usually in better model convergence, but much slower training.

• num_workers (int or 'auto' (default = 0)) – Number of parallel workers that will load the data. Set ‘auto’ to let RSP choose the optimal number of workers, set 0 to disable multiprocessing. Can increase training speed, but can also cause errors (e.g. pickling errors).

• classification (bool (default = None)) – If True then perform classification (e.g. semantic segmentation) task, else perform regression task. If not defined then is read from from train dataset.

• num_classes (int (optional)) – Number of classes for classification task. If not defined then is read from train dataset.

• y_nodata (int or float (optional)) – You can define which value in y raster corresponds to nodata and areas that contain nodata in y raster will be ignored while training and testing. If not defined then is read from train dataset.

• **kwargs – Additional keyword arguments that are used to initialise model. They are different for every model, so read the documentation.

Returns:

Trained model.

Return type:

torch.nn model or SklearnModel

Examples

>>> x_out, y_out = rsp.segmentation.generate_tiles(
...     x,
...     y,
...     tile_size=256,
...     shuffle=True,
...     split=[3, 1, 1],
...     split_names=['train', 'val', 'test']
...     )
>>> train_ds = [x_out, y_out[0], 'train']
>>> val_ds = [x_out, y_out[0], 'val']
>>> model = rsp.segmentation.train(
...     train_ds,
...     val_ds,
...     model='UperNet',
...     backbone='ConvNeXTV2',
...     model_file='/home/rsp_test/model/upernet.ckpt',
...     epochs=100,
...     batch_size=32
... )
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
  | Name    | Type                           | Params
-----------------------------------------------------------
0 | model   | UperNetForSemanticSegmentation | 59.8 M
1 | loss_fn | CrossEntropyLoss               | 0
-----------------------------------------------------------
59.8 M    Trainable params
0         Non-trainable params
59.8 M    Total params
239.395   Total estimated model params size (MB)
Epoch 9: 100% #############################################
223/223 [1:56:20<00:00, 31.30s/it, v_num=54,
train_loss_step=0.326, train_acc_step=0.871, train_auroc_step=0.796, train_iou_step=0.655,
val_loss_step=0.324, val_acc_step=0.869, val_auroc_step=0.620, val_iou_step=0.678,
val_loss_epoch=0.334, val_acc_epoch=0.807, val_auroc_epoch=0.795, val_iou_epoch=0.688,
train_loss_epoch=0.349, train_acc_epoch=0.842, train_auroc_epoch=0.797, train_iou_epoch=0.648]
`Trainer.fit` stopped: `max_epochs=10` reached.

>>> x_mo = '/home/rsp_test/model/x_montana.zarr'
>>> y_mo = '/home/rsp_test/model/y_montana.zarr'
>>> x_id = '/home/rsp_test/model/x_idaho.zarr'
>>> y_id = '/home/rsp_test/model/y_idaho.zarr'
>>> # Training on two different datasets - one from Montana and one from Idaho
>>> train_datasets = [[x_mo, y_mo, ['area_1', 'area_2']], [x_id, y_id, ['area_3', 'area_6', 'area8']]]
>>> val_datasets = [[x_mo, y_mo, ['area_3', 'area_4']], [x_id, y_id, ['area_1']]]
>>> model = rsp.segmentation.train(
...     train_datasets,
...     val_datasets,
...     model='UperNet',
...     backbone='ConvNeXTV2',
...     model_file='/home/rsp_test/model/upernet.ckpt',
...     epochs=100,
...     batch_size=32
... )
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
  | Name    | Type                           | Params
-----------------------------------------------------------
0 | model   | UperNetForSemanticSegmentation | 59.8 M
1 | loss_fn | CrossEntropyLoss               | 0
-----------------------------------------------------------
59.8 M    Trainable params
0         Non-trainable params
59.8 M    Total params
239.395   Total estimated model params size (MB)
Epoch 99: 100% #############################################
223/223 [1:56:20<00:00, 31.30s/it, v_num=54, train_loss_step=0.326,
train_acc_step=0.871, train_auroc_step=0.796, train_iou_step=0.655,
val_loss_step=0.324, val_acc_step=0.869, val_auroc_step=0.620, val_iou_step=0.678,
val_loss_epoch=0.334, val_acc_epoch=0.807, val_auroc_epoch=0.795, val_iou_epoch=0.688,
train_loss_epoch=0.349, train_acc_epoch=0.842, train_auroc_epoch=0.797, train_iou_epoch=0.648]
`Trainer.fit` stopped: `max_epochs=100` reached.

List of available NN models

Model	Backbone	Classification	Regression	No metrics-related freeze issues	No convergence issues	Fine-tuning available	Reference
BEiT		+	-[4]	+	+	+	Huggingface Transformers
ConditionalDETR		+	+	+-[1]	+	Not tested	Huggingface Transformers
Data2Vec		+	-[4]	+	+	+	Huggingface Transformers
DETR		+	+	+-[1]	+	+	Huggingface Transformers
DPT		+	-[4]	+	+	+	Huggingface Transformers
Mask2Former		+	-[4]	+-[1]	+-[2]	+	Huggingface Transformers
MaskFormer		+	-[4]	+	+-[2]		Huggingface Transformers
MobileNetV2		+	+	+	+	+	Huggingface Transformers
MobileViT		+	+	+	+	+	Huggingface Transformers
MobileViTV2		+	+	+	+	+	Huggingface Transformers
OneFormer	Swin	+	-[4]	+-[1]	+-[2]	+	Huggingface Transformers
	ConvNeXT	+	-[4]	+-[1]	+-[2]	Not tested
	ConvNeXTV2	+	-[4]	+-[1]	+-[2]	Not tested
	DiNAT	-[3]	-[3]
SegFormer		+	-[4]	+	+	+	Huggingface Transformers
UperNet	Swin	+	-[4]	+	+	+	Huggingface Transformers
	ResNet	+	+	+	+	Not tested
	ConvNeXT	+	+	+	+	+
	ConvNeXTV2	+	+	+	+	Not tested
DeepLabV3	MobileNet_V3_Large	+	+	+	+	+	Torchvision
	ResNet50	+	+	+	+	+	Torchvision
	ResNet101	+	+	+	+	+	Torchvision
FCN	ResNet50	+	+	+	+	+	Torchvision
	ResNet101	+	+	+	+	+	Torchvision
LRASPP		+	+	+	+	+	Torchvision

You can fine-tune pre-trained model by defining weights. For models from Transformers you can get available weights from Huggingface Hub, for Torchvision models you just set weights = True.

rsp.segmentation.train also saves CSV and Tensorboard logs in directory where checkpoint file is saved.

List of available Scikit-learn models

Model	Kernel/solver [5]	Classification	Regression	Warm start	Reference
Nearest Neighbors		+	+	-	Scikit-learn
Logistic Regression	lbfgs	+	-	+	Scikit-learn
	liblinear	+	-	-
	newton-cg	+	-	+
	newton-cholesky	+	-	+
	sag	+	-	+
	saga	+	-	+
Ridge		-	+	-	Scikit-learn
Lasso		-	+	-	Scikit-learn
ElasticNet		-	+	-	Scikit-learn
SVM	rbf	+	+	-	Scikit-learn
	linear	+	+	-
	poly	+	+	-
	sigmoid	+	+	-
Gaussian Process		+	+	+	Scikit-learn
Naive Bayes		+	-	-	Scikit-learn
QDA		+	-	-	Scikit-learn
Decision Tree		+	+	-	Scikit-learn
Random Forest		+	+	+	Scikit-learn
AdaBoost		+	+	-	Scikit-learn
Gradient Boosting		+	+	+	Scikit-learn
Multilayer Perceptron		+	+	+	Scikit-learn
XGBoost		+	+	-	XGBoost
XGBoost Random Forest		+	+	-	XGBoost

Models that support warm start can be fine-tuned using pre-trained models with checkpoint argument.

Some models can have issues while saving, especially when trained on big datasets. Some models (like SVM) can train for a very long time or (like Gaussian process) can have memory issues with big datasets. So we recommend using Scikit-learn models only for small datasets.

[1] (1,2,3,4,5,6)

These models can freeze forever on one of first steps. It happens due to some confusion matrix-based metrics related error. Try to restart training with less_metrics = True.

[2] (1,2,3,4,5)

These models showed very poor performance on tests. They only converged slowly with lr=1e-5, but much more slow than other models. I do not recommend using them.

[3] (1,2)

DiNAT requires natten library, that is not available on Windows and Mac and not available via Conda. RSP supports DiNAT backbone, but you need to install natten in your python env manually.

[4] (1,2,3,4,5,6,7,8,9,10)

Loss becomes nan while training several transformers if the task is regression.

[5]

Is defined with backbone argument.