Training Custom Plantseg Model (Pytorch)#
This blog post aims to help biologists that are not experienced in programming and deep learning to train their own model for PlantSeg. PlantSeg is a versatile 3D segmentation tool of data with fluorescently labelled membranes. Originally it was developed for segmentation of dense plant tissues at cellular resolution; however, it can be applied to a wide variety of data. Plantseg combines deep learning based prediction of fluorescently labelled membranes and graph partitioning methods to get final instance segmentation. You can read more about the inner workings of the method in the original publication. There are a few pre-trained models provided, and you can easily do the segmentation using these models in the GUI. You can find the instructions on how to use the GUI in the original repository. However, training a new model is not possible in the GUI. The training is based on the pytorch-3dunet repository. This blog post tries to simplify the process of training a deep learning model for Plantseg by providing instructions on how to do it from the jupyter notebook, which is a more user-friendly way than the command line. You can also use these instructions to tune the pre-trained model with your data instead of performing the training from scratch.
This blog post is for you if:#
You have already tried to predict with provided pre-trained models, and felt like results were good but not good enough for your data because your data differs substantially from plant data the models were trained on.
You have thought about training your own model, however, feel intimidated by the instructions in the pytorch-3dunet repository on GitHub because originally the training is done from the terminal with YML configuration files.
Custom training Plantseg model is not for you if:#
You do not have good quality annotated data (so-called ground truth data).
You have very little data.
You don’t have access to the hardware needed to handle your data and deep learning model’s training.
What will you need?#
A conda/mamba environment or a singularity container with required packages (described in Step 1). If you don’t have yet conda/mamba installed on your machine, you can read this blog post by Mara on how to set it up. Or if you want to do the training on the HPC cluster you can read this blog post by Till on how to get started with it.
Annotated (labelled) data, where labels are of good quality, otherwise, the model will learn bad “knowledge”.
Quite a bit of data if you are training from scratch because deep learning models are data greedy. Therefore, you need at least a couple of hundred of 3D timelapse time points. The model needs to “see” a variety of images during training to “learn” well, that means to perform later on well on “unseen” data.
Hardware powerful enough for deep learning, including NVIDIA GPU. This will also depend on your data and parameters that you choose for the training, particularly patch size, which determines how big overlapping patches will be extracted from your images. One patch should fit into your GPU memory but it should not be way smaller than objects in your images. Therefore, it might be that a laptop with a GPU of 4 or 8 GB will be enough for the training but it also can happen that you will need to do the training on the workstation or HPC cluster.
Note: The training in this blog post example was done on the Taurus cluster at TU Dresden with 4 NVIDIA A100 GPUs.
Step 1: Prepare the Environment/container#
If you are using conda/mamba to create the environment, the easiest way to install required packages is via the following command:
mamba create -n plantseg-env -c pytorch -c conda-forge -c lcerrone -c awolny plantseg
conda activate pytorch3dunet
Pay attention that this command will install the newest available versions of libraries (or pinned versions in some cases). Depending on your GPU, you might need a different version of cuda for pytorch to be able to run on the GPU and not only CPU, which is very important for speed of the training. You can check which versions you would need here, and follow installation instructions accordingly.
Note: If you want to train the model on the alpha partition of the Taurus cluster at TUD, the versions that you need to install can be found here.
To check whether you have all packages that we will need installed, run the following cell and see whether it executes without any errors.
from pytorch3dunet.train import main as train_plantseg
from natsort import natsorted
from glob import glob
from pathlib import Path
from sklearn.model_selection import train_test_split
import h5py
import os
import numpy as np
from tqdm import tqdm
from skimage.io import imread
from tifffile import imwrite
import yaml
import random
import torch
from pytorch3dunet.unet3d.config import load_config
from pytorch3dunet.unet3d.trainer import create_trainer
from pytorch3dunet.unet3d.utils import get_logger
/app/env/lib/python3.10/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
Step 2: Split your Data to Train, Validation and Test Sets#
Typically, in deep learning model’s training workflow you split your data (raw and labels data) into three parts: training, validation and testing. To put it in very simple words, the training will be done on the training data, validation data is used to assess how well the model is doing up to that point and to adjust the direction it will continue the training, and testing data is used to evaluate the final model after it finished training. Testing data was not “seen” by the model during the training, therefore, it can be used for the evaluation.
First of all, we give the directories containing multiple 3D tif images (timepoints of the timelapse) of the raw data and labels data, and load all the data to memory (so it needs to fit into RAM) because we will split it to the before-mentioned three sets.
# give the path to the folder containing raw data
trn_raw_dir = "/projects/p_bioimage/laura/data/raw_timepoints"
# get all files with the extension .tif from the raw data path
filenames = natsorted(glob(os.path.join(trn_raw_dir, "*.tif")))
# read all images
train_images = [imread(os.path.join(trn_raw_dir, f)) for f in tqdm(filenames)]
100%|██████████| 345/345 [02:12<00:00, 2.61it/s]
# do the same for the labels data
trn_labels_dir = "/projects/p_bioimage/laura/data/labels_timepoints"
filenames_lbl = natsorted(glob(os.path.join(trn_labels_dir, "*.tif")))
train_labels = [imread(os.path.join(trn_labels_dir, f)) for f in tqdm(filenames_lbl)]
100%|██████████| 345/345 [02:32<00:00, 2.27it/s]
We will do the splitting the following way:
Training data 70%
Validation data 20%
Testing data 10%
# set aside 10% of the whole dataset for the evaluation (test set)
X_train, X_test, Y_train, Y_test = train_test_split(train_images, train_labels, test_size=0.1, shuffle = True, random_state = 8)
# freeing up some memory before continuing
del train_images
del train_labels
# use the same function as above for the split into training and validation sets
X_train, X_val, Y_train, Y_val = train_test_split(X_train, Y_train, test_size=0.28, random_state= 8)
# let's write a small function that we can just call to save the images
def save_images(images, path):
# make a new directory, including all parent directories, if it does not exist yet (must have write access)
Path(path).mkdir(parents=True, exist_ok=True)
# iterate through images and save them
for i, img in tqdm(enumerate(images), total=len(images)):
imwrite(os.path.join(path, str(i) + ".tif"), img)
# call the function multiple times
# save training data
save_images(X_train, "data/train/raw")
save_images(Y_train, "data/train/labels")
# save validation data
save_images(X_val, "data/val/raw")
save_images(Y_val, "data/val/labels")
# save testing data
save_images(X_test, "data/test/raw")
save_images(Y_test, "data/test/labels")
100%|██████████| 223/223 [00:46<00:00, 4.80it/s]
100%|██████████| 223/223 [01:31<00:00, 2.44it/s]
100%|██████████| 87/87 [00:18<00:00, 4.72it/s]
100%|██████████| 87/87 [00:35<00:00, 2.43it/s]
100%|██████████| 35/35 [00:07<00:00, 4.78it/s]
100%|██████████| 35/35 [00:14<00:00, 2.43it/s]
Step 3: Convert Data to HDF5 format#
This particular implementation of deep learning model’s training requires the dataset to be in HDF5 format.
def convert_tif_to_hdf5(trn_raw_dir, trn_labels_dir, val_raw_dir, val_labels_dir, save_dir):
# MAKE TRAIN DATASET
# First load only filenames of raw training data. Corresponding labels should have matching filenames.
# This way we can iterate over a list of filenames, and load only one file into memory during each iteration.
print(f"Loading all .tif filenames from {trn_raw_dir}")
fetched_files = natsorted(glob(os.path.join(trn_raw_dir, "*.tif")))
filenames = [Path(f).name for f in fetched_files]
# Creating a "train" folder in the given save directory
Path(os.path.join(save_dir, "train")).mkdir(parents=True, exist_ok=True)
for tif_name in tqdm(filenames, desc="Converting .tif files to hdf5. Train dataset"):
raw_image = imread(os.path.join(trn_raw_dir, tif_name)) # load raw image
labels_image = imread(os.path.join(trn_labels_dir, tif_name)) # load labels image
hf = h5py.File(os.path.join(save_dir, "train", tif_name.split(".")[0] + ".hdf5"), 'a') # open a hdf5 file
hf.create_dataset(name="raw", data=np.array(raw_image)) # write raw data to hdf5 file
hf.create_dataset(name="label", data=np.array(labels_image)) # write label data to hdf5 file
hf.close() # close the hdf5 file
# MAKE VALIDATION DATASET
print(f"Loading all .tif filenames from {val_raw_dir}")
fetched_files_val = natsorted(glob(os.path.join(val_raw_dir, "*.tif")))
filenames_val = [Path(f).name for f in fetched_files_val]
Path(os.path.join(save_dir, "val")).mkdir(parents=True, exist_ok=True)
for tif_name in tqdm(filenames_val, desc="Converting .tif files to hdf5. Val dataset"):
raw_image = imread(os.path.join(val_raw_dir, tif_name)) # load raw image
labels_image = imread(os.path.join(val_labels_dir, tif_name)) # load labels image
hf = h5py.File(os.path.join(save_dir, "val", tif_name.split(".")[0] + ".hdf5"), 'a') # open a hdf5 file
hf.create_dataset(name="raw", data=np.array(raw_image)) # write raw data to hdf5 file
hf.create_dataset(name="label", data=np.array(labels_image)) # write label data to hdf5 file
hf.close()
trn_raw_dir = "/projects/p_bioimage/laura/data/train/raw"
trn_labels_dir = "/projects/p_bioimage/laura/data/train/labels"
val_raw_dir = "/projects/p_bioimage/laura/data/val/raw"
val_labels_dir = "/projects/p_bioimage/laura/data/val/labels"
save_dir = "data/hdf5"
convert_tif_to_hdf5(trn_raw_dir, trn_labels_dir, val_raw_dir, val_labels_dir, save_dir)
Loading all .tif filenames from /projects/p_bioimage/laura/data/val/raw
Converting .tif files to hdf5. Val dataset: 100%|██████████| 87/87 [02:09<00:00, 1.49s/it]
Open Training Configuration file#
with open("/projects/p_bioimage/laura/train_config.yml", 'r') as file:
config = yaml.safe_load(file)
config
{'model': {'name': 'ResidualUNet3D',
'in_channels': 1,
'out_channels': 1,
'layer_order': 'gcr',
'f_maps': 32,
'num_groups': 8,
'final_sigmoid': True},
'loss': {'name': 'BCEDiceLoss',
'ignore_index': None,
'skip_last_target': True},
'optimizer': {'learning_rate': 0.0002, 'weight_decay': 1e-05},
'eval_metric': {'name': 'BoundaryAdaptedRandError',
'threshold': 0.4,
'use_last_target': True,
'use_first_input': True},
'lr_scheduler': {'name': 'ReduceLROnPlateau',
'mode': 'min',
'factor': 0.2,
'patience': 20},
'trainer': {'eval_score_higher_is_better': False,
'checkpoint_dir': 'CHECKPOINT_DIR',
'resume': None,
'pre_trained': None,
'validate_after_iters': 1000,
'log_after_iters': 500,
'max_num_epochs': 1000,
'max_num_iterations': 150000},
'loaders': {'num_workers': 8,
'raw_internal_path': '/raw',
'label_internal_path': '/label',
'train': {'file_paths': ['PATH_TO_TRAIN_DIR'],
'slice_builder': {'name': 'FilterSliceBuilder',
'patch_shape': [80, 170, 170],
'stride_shape': [20, 40, 40],
'threshold': 0.6,
'slack_acceptance': 0.01},
'transformer': {'raw': [{'name': 'Standardize'},
{'name': 'RandomFlip'},
{'name': 'RandomRotate90'},
{'name': 'RandomRotate',
'axes': [[2, 1]],
'angle_spectrum': 45,
'mode': 'reflect'},
{'name': 'ElasticDeformation', 'spline_order': 3},
{'name': 'GaussianBlur3D', 'execution_probability': 0.5},
{'name': 'AdditiveGaussianNoise', 'execution_probability': 0.2},
{'name': 'AdditivePoissonNoise', 'execution_probability': 0.2},
{'name': 'ToTensor', 'expand_dims': True}],
'label': [{'name': 'RandomFlip'},
{'name': 'RandomRotate90'},
{'name': 'RandomRotate',
'axes': [[2, 1]],
'angle_spectrum': 45,
'mode': 'reflect'},
{'name': 'ElasticDeformation', 'spline_order': 0},
{'name': 'StandardLabelToBoundary', 'append_label': True},
{'name': 'ToTensor', 'expand_dims': False}]}},
'val': {'file_paths': ['PATH_TO_VAL_DIR'],
'slice_builder': {'name': 'FilterSliceBuilder',
'patch_shape': [80, 170, 170],
'stride_shape': [80, 170, 170],
'threshold': 0.6,
'slack_acceptance': 0.01},
'transformer': {'raw': [{'name': 'Standardize'},
{'name': 'ToTensor', 'expand_dims': True}],
'label': [{'name': 'StandardLabelToBoundary', 'append_label': True},
{'name': 'ToTensor', 'expand_dims': False}]}}}}
config["loaders"]["train"]["file_paths"] = ["/projects/p_bioimage/laura/data/hdf5_for_plantseg/train"]
print(config["loaders"]["train"]["file_paths"])
['/projects/p_bioimage/laura/data/hdf5_for_plantseg/train']
config["loaders"]["val"]["file_paths"] = ["/projects/p_bioimage/laura/data/hdf5_for_plantseg/val"]
print(config["loaders"]["val"]["file_paths"])
['/projects/p_bioimage/laura/data/hdf5_for_plantseg/val']
config["model"]["name"] = "UNet3D"
config["lr_scheduler"]["patience"] = 10
config["trainer"]["validate_after_iters"] = 100
config["trainer"]["log_after_iters"] = 100
config["trainer"]["max_num_epochs"] = 500
config["trainer"]["max_num_iterations"] = 60000
config["optimizer"]["learning_rate"] = 0.001
config["trainer"]["checkpoint_dir"] = "Model_3"
# config["loaders"]["batch_size"] = 2
Patch size#
In this case we also need to change patch size, because it needs to be smaller than data size.
# print patch size in the default configuration
print(config["loaders"]["train"]["slice_builder"]["patch_shape"])
[80, 170, 170]
config["loaders"]["train"]["slice_builder"]["patch_shape"] = [64, 260, 260]
print(config["loaders"]["train"]["slice_builder"]["patch_shape"])
[64, 260, 260]
Stride shape#
We also change the stride shape, which determines how much our patches will overlap
STRIDE_MENU = {
"accurate": 0.5,
"balanced": 0.75,
"draft": 0.9
}
def get_stride_shape(patch_shape, stride_key):
return [max(int(p * STRIDE_MENU[stride_key]), 1) for p in patch_shape]
# print patch stride in the default configuration
print(config["loaders"]["train"]["slice_builder"]["stride_shape"])
[20, 40, 40]
config["loaders"]["train"]["slice_builder"]["stride_shape"] = get_stride_shape(config["loaders"]["train"]["slice_builder"]["patch_shape"], "accurate")
config["loaders"]["train"]["slice_builder"]["stride_shape"] = [1, 130, 130]
#print(config["loaders"]["train"]["slice_builder"]["stride_shape"])
Now we should do the same for the validation dataset
config["loaders"]["val"]["slice_builder"]["stride_shape"] = config["loaders"]["train"]["slice_builder"]["stride_shape"]
config["loaders"]["val"]["slice_builder"]["patch_shape"] = config["loaders"]["train"]["slice_builder"]["patch_shape"]
print(config["loaders"]["num_workers"])
8
config["loaders"]["num_workers"] = 6
Continuing Training the Pre-trained Model#
If a pretrained model worked relatively well to your data, it might make sense to take pretrained model’s weights and continue training your model on your data, which will be less time constly. If you want to do the training this way, download pre-trained model and provide the path to it to the parameter below:
# cfg["trainer"]["checkpoint_dir"] = [""]
config["trainer"]["checkpoint_dir"]
'Model_3'
config["device"] = None
config
{'model': {'name': 'UNet3D',
'in_channels': 1,
'out_channels': 1,
'layer_order': 'gcr',
'f_maps': 32,
'num_groups': 8,
'final_sigmoid': True},
'loss': {'name': 'BCEDiceLoss',
'ignore_index': None,
'skip_last_target': True},
'optimizer': {'learning_rate': 0.001, 'weight_decay': 1e-05},
'eval_metric': {'name': 'BoundaryAdaptedRandError',
'threshold': 0.4,
'use_last_target': True,
'use_first_input': True},
'lr_scheduler': {'name': 'ReduceLROnPlateau',
'mode': 'min',
'factor': 0.2,
'patience': 10},
'trainer': {'eval_score_higher_is_better': False,
'checkpoint_dir': 'Model_3',
'resume': None,
'pre_trained': None,
'validate_after_iters': 100,
'log_after_iters': 100,
'max_num_epochs': 500,
'max_num_iterations': 60000},
'loaders': {'num_workers': 6,
'raw_internal_path': '/raw',
'label_internal_path': '/label',
'train': {'file_paths': ['/projects/p_bioimage/laura/data/hdf5_for_plantseg/train'],
'slice_builder': {'name': 'FilterSliceBuilder',
'patch_shape': [64, 260, 260],
'stride_shape': [1, 130, 130],
'threshold': 0.6,
'slack_acceptance': 0.01},
'transformer': {'raw': [{'name': 'Standardize'},
{'name': 'RandomFlip'},
{'name': 'RandomRotate90'},
{'name': 'RandomRotate',
'axes': [[2, 1]],
'angle_spectrum': 45,
'mode': 'reflect'},
{'name': 'ElasticDeformation', 'spline_order': 3},
{'name': 'GaussianBlur3D', 'execution_probability': 0.5},
{'name': 'AdditiveGaussianNoise', 'execution_probability': 0.2},
{'name': 'AdditivePoissonNoise', 'execution_probability': 0.2},
{'name': 'ToTensor', 'expand_dims': True}],
'label': [{'name': 'RandomFlip'},
{'name': 'RandomRotate90'},
{'name': 'RandomRotate',
'axes': [[2, 1]],
'angle_spectrum': 45,
'mode': 'reflect'},
{'name': 'ElasticDeformation', 'spline_order': 0},
{'name': 'StandardLabelToBoundary', 'append_label': True},
{'name': 'ToTensor', 'expand_dims': False}]}},
'val': {'file_paths': ['/projects/p_bioimage/laura/data/hdf5_for_plantseg/val'],
'slice_builder': {'name': 'FilterSliceBuilder',
'patch_shape': [64, 260, 260],
'stride_shape': [1, 130, 130],
'threshold': 0.6,
'slack_acceptance': 0.01},
'transformer': {'raw': [{'name': 'Standardize'},
{'name': 'ToTensor', 'expand_dims': True}],
'label': [{'name': 'StandardLabelToBoundary', 'append_label': True},
{'name': 'ToTensor', 'expand_dims': False}]}}},
'device': None}
Training#
device_str = config.get('device', None)
if device_str is not None:
print(f"Device specified in config: '{device_str}'")
if device_str.startswith('cuda') and not torch.cuda.is_available():
print('CUDA not available, using CPU')
device_str = 'cpu'
else:
device_str = "cuda:0" if torch.cuda.is_available() else 'cpu'
print(f"Using '{device_str}' device")
device = torch.device(device_str)
config['device'] = device
Using 'cuda:0' device
manual_seed = config.get('manual_seed', None)
if manual_seed is not None:
logger.info(f'Seed the RNG for all devices with {manual_seed}')
logger.warning('Using CuDNN deterministic setting. This may slow down the training!')
random.seed(manual_seed)
torch.manual_seed(manual_seed)
# see https://pytorch.org/docs/stable/notes/randomness.html
torch.backends.cudnn.deterministic = True
# create trainer
trainer = create_trainer(config)
# Start training
trainer.fit()
2023-02-03 11:50:50,569 [MainThread] INFO UNet3DTrainer - Using 4 GPUs for training
2023-02-03 11:50:50,570 [MainThread] INFO UNet3DTrainer - Sending the model to 'cuda:0'
2023-02-03 11:50:54,192 [MainThread] INFO UNet3DTrainer - Number of learnable params 4081267
2023-02-03 11:50:54,193 [MainThread] INFO Dataset - Creating training and validation set loaders...
2023-02-03 11:50:54,193 [MainThread] WARNING Dataset - Cannot find dataset class in the config. Using default 'StandardHDF5Dataset'.
2023-02-03 11:50:54,390 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/0.hdf5...
2023-02-03 11:50:54,972 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:50:55,403 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:50:55,404 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/1.hdf5...
2023-02-03 11:50:55,893 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:50:56,314 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:50:56,316 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/2.hdf5...
2023-02-03 11:50:56,758 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:50:57,161 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:50:57,162 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/3.hdf5...
2023-02-03 11:50:57,665 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:50:58,076 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:50:58,077 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/4.hdf5...
2023-02-03 11:50:58,719 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:50:59,129 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:50:59,130 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/5.hdf5...
2023-02-03 11:50:59,688 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:00,099 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:00,100 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/6.hdf5...
2023-02-03 11:51:00,721 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:01,141 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:01,142 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/7.hdf5...
2023-02-03 11:51:01,641 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:02,052 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:02,052 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/8.hdf5...
2023-02-03 11:51:02,457 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:02,873 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:02,874 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/9.hdf5...
2023-02-03 11:51:03,372 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:03,781 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:03,782 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/10.hdf5...
2023-02-03 11:51:04,261 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:04,671 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:04,672 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/11.hdf5...
2023-02-03 11:51:05,403 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:05,805 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:05,806 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/12.hdf5...
2023-02-03 11:51:06,440 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:06,852 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:06,852 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/13.hdf5...
2023-02-03 11:51:07,327 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:07,743 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:07,744 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/14.hdf5...
2023-02-03 11:51:08,556 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:08,970 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:08,971 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/15.hdf5...
2023-02-03 11:51:09,727 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:10,145 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:10,146 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/16.hdf5...
2023-02-03 11:51:10,595 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:11,012 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:11,013 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/17.hdf5...
2023-02-03 11:51:11,425 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:11,844 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:11,845 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/18.hdf5...
2023-02-03 11:51:12,276 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:12,697 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:12,698 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/19.hdf5...
2023-02-03 11:51:13,602 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:14,013 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:14,014 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/20.hdf5...
2023-02-03 11:51:14,547 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:14,962 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:14,963 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/21.hdf5...
2023-02-03 11:51:15,398 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:15,811 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:15,812 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/22.hdf5...
2023-02-03 11:51:16,759 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:17,171 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:17,172 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/23.hdf5...
2023-02-03 11:51:17,657 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:18,072 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:18,072 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/24.hdf5...
2023-02-03 11:51:18,514 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:18,923 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:18,924 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/25.hdf5...
2023-02-03 11:51:19,480 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:19,885 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:19,886 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/26.hdf5...
2023-02-03 11:51:20,290 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:20,709 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:20,710 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/27.hdf5...
2023-02-03 11:51:21,121 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:21,532 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:21,533 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/28.hdf5...
2023-02-03 11:51:22,037 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:22,451 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:22,452 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/29.hdf5...
2023-02-03 11:51:22,870 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:23,287 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:23,288 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/30.hdf5...
2023-02-03 11:51:23,754 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:24,168 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:24,169 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/31.hdf5...
2023-02-03 11:51:24,546 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:24,959 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:24,959 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/32.hdf5...
2023-02-03 11:51:25,716 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:26,115 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:26,116 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/33.hdf5...
2023-02-03 11:51:26,848 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:27,261 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:27,262 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/34.hdf5...
2023-02-03 11:51:27,820 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:28,227 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:28,228 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/35.hdf5...
2023-02-03 11:51:28,662 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:29,078 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:29,079 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/36.hdf5...
2023-02-03 11:51:29,640 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:30,050 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:30,051 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/37.hdf5...
2023-02-03 11:51:30,713 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:31,115 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:31,116 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/38.hdf5...
2023-02-03 11:51:31,743 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:32,152 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:32,153 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/39.hdf5...
2023-02-03 11:51:32,658 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:33,073 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:33,074 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/40.hdf5...
2023-02-03 11:51:33,664 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:34,075 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:34,076 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/41.hdf5...
2023-02-03 11:51:34,625 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:35,039 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:35,039 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/42.hdf5...
2023-02-03 11:51:35,599 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:36,009 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:36,010 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/43.hdf5...
2023-02-03 11:51:36,517 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:36,928 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:36,929 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/44.hdf5...
2023-02-03 11:51:38,193 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:38,596 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:38,597 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/45.hdf5...
2023-02-03 11:51:39,443 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:39,853 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:39,854 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/46.hdf5...
2023-02-03 11:51:40,414 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:40,830 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:40,831 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/47.hdf5...
2023-02-03 11:51:41,370 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:41,782 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:41,783 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/48.hdf5...
2023-02-03 11:51:42,470 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:42,884 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:42,885 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/49.hdf5...
2023-02-03 11:51:43,348 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:43,755 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:43,756 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/50.hdf5...
2023-02-03 11:51:44,307 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:44,714 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:44,715 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/51.hdf5...
2023-02-03 11:51:45,279 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:45,688 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:45,689 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/52.hdf5...
2023-02-03 11:51:46,166 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:46,576 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:46,577 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/53.hdf5...
2023-02-03 11:51:47,154 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:47,560 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:47,561 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/54.hdf5...
2023-02-03 11:51:48,004 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:48,414 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:48,415 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/55.hdf5...
2023-02-03 11:51:48,902 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:49,310 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:49,311 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/56.hdf5...
2023-02-03 11:51:50,205 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:50,606 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:50,607 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/57.hdf5...
2023-02-03 11:51:51,353 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:51,766 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:51,767 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/58.hdf5...
2023-02-03 11:51:52,207 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:52,621 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:52,622 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/59.hdf5...
2023-02-03 11:51:53,158 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:53,577 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:53,578 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/60.hdf5...
2023-02-03 11:51:54,378 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:54,790 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:54,791 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/61.hdf5...
2023-02-03 11:51:55,464 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:55,877 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:55,878 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/62.hdf5...
2023-02-03 11:51:56,239 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:56,652 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:56,653 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/63.hdf5...
2023-02-03 11:51:57,218 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:57,630 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:57,631 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/64.hdf5...
2023-02-03 11:51:58,129 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:58,542 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:58,543 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/65.hdf5...
2023-02-03 11:51:59,352 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:51:59,758 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:51:59,759 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/66.hdf5...
2023-02-03 11:52:00,206 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:00,623 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:00,623 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/67.hdf5...
2023-02-03 11:52:01,111 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:01,527 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:01,528 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/68.hdf5...
2023-02-03 11:52:01,977 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:02,378 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:02,379 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/69.hdf5...
2023-02-03 11:52:03,246 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:03,667 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:03,668 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/70.hdf5...
2023-02-03 11:52:04,213 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:04,633 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:04,634 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/71.hdf5...
2023-02-03 11:52:05,182 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:05,590 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:05,591 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/72.hdf5...
2023-02-03 11:52:06,212 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:06,628 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:06,629 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/73.hdf5...
2023-02-03 11:52:07,140 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:07,553 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:07,555 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/74.hdf5...
2023-02-03 11:52:08,230 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:08,639 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:08,640 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/75.hdf5...
2023-02-03 11:52:09,207 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:09,614 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:09,615 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/76.hdf5...
2023-02-03 11:52:10,323 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:10,738 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:10,739 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/77.hdf5...
2023-02-03 11:52:11,304 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:11,717 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:11,718 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/78.hdf5...
2023-02-03 11:52:12,164 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:12,580 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:12,581 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/79.hdf5...
2023-02-03 11:52:13,182 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:13,592 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:13,593 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/80.hdf5...
2023-02-03 11:52:14,446 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:14,860 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:14,861 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/81.hdf5...
2023-02-03 11:52:15,415 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:15,830 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:15,831 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/82.hdf5...
2023-02-03 11:52:16,252 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:16,670 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:16,671 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/83.hdf5...
2023-02-03 11:52:17,137 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:17,544 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:17,545 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/84.hdf5...
2023-02-03 11:52:18,038 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:18,457 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:18,458 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/85.hdf5...
2023-02-03 11:52:19,417 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:19,819 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:19,820 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/86.hdf5...
2023-02-03 11:52:20,342 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:20,754 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:20,755 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/87.hdf5...
2023-02-03 11:52:21,294 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:21,709 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:21,710 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/88.hdf5...
2023-02-03 11:52:22,209 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:22,627 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:22,628 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/89.hdf5...
2023-02-03 11:52:23,130 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:23,544 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:23,545 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/90.hdf5...
2023-02-03 11:52:24,129 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:24,539 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:24,540 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/91.hdf5...
2023-02-03 11:52:25,016 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:25,419 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:25,420 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/92.hdf5...
2023-02-03 11:52:26,180 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:26,596 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:26,597 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/93.hdf5...
2023-02-03 11:52:27,095 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:27,511 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:27,512 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/94.hdf5...
2023-02-03 11:52:28,088 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:28,491 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:28,491 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/95.hdf5...
2023-02-03 11:52:29,037 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:29,458 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:29,459 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/96.hdf5...
2023-02-03 11:52:30,175 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:30,587 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:30,588 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/97.hdf5...
2023-02-03 11:52:31,072 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:31,484 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:31,485 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/98.hdf5...
2023-02-03 11:52:32,025 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:32,440 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:32,441 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/99.hdf5...
2023-02-03 11:52:32,934 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:33,346 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:33,347 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/100.hdf5...
2023-02-03 11:52:33,724 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:34,133 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:34,134 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/101.hdf5...
2023-02-03 11:52:34,777 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:35,177 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:35,178 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/102.hdf5...
2023-02-03 11:52:35,732 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:36,137 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:36,138 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/103.hdf5...
2023-02-03 11:52:36,621 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:37,034 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:37,035 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/104.hdf5...
2023-02-03 11:52:37,560 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:37,961 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:37,962 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/105.hdf5...
2023-02-03 11:52:38,373 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:38,785 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:38,786 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/106.hdf5...
2023-02-03 11:52:39,219 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:39,628 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:39,629 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/107.hdf5...
2023-02-03 11:52:40,443 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:40,847 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:40,848 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/108.hdf5...
2023-02-03 11:52:41,649 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:42,068 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:42,069 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/109.hdf5...
2023-02-03 11:52:42,645 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:43,050 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:43,051 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/110.hdf5...
2023-02-03 11:52:43,593 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:44,001 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:44,002 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/111.hdf5...
2023-02-03 11:52:44,531 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:44,930 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:44,931 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/112.hdf5...
2023-02-03 11:52:45,462 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:45,869 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:45,870 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/113.hdf5...
2023-02-03 11:52:46,475 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:46,875 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:46,876 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/114.hdf5...
2023-02-03 11:52:47,352 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:47,756 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:47,757 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/115.hdf5...
2023-02-03 11:52:48,244 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:48,652 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:48,653 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/116.hdf5...
2023-02-03 11:52:49,544 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:49,948 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:49,949 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/117.hdf5...
2023-02-03 11:52:50,562 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:50,972 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:50,973 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/118.hdf5...
2023-02-03 11:52:51,381 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:51,779 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:51,780 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/119.hdf5...
2023-02-03 11:52:52,371 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:52,780 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:52,781 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/120.hdf5...
2023-02-03 11:52:53,530 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:53,946 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:53,947 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/121.hdf5...
2023-02-03 11:52:54,401 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:54,814 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:54,815 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/122.hdf5...
2023-02-03 11:52:55,327 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:55,740 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:55,740 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/123.hdf5...
2023-02-03 11:52:56,183 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:56,592 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:56,593 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/124.hdf5...
2023-02-03 11:52:57,102 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:57,506 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:57,507 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/125.hdf5...
2023-02-03 11:52:57,991 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:58,407 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:58,408 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/126.hdf5...
2023-02-03 11:52:58,840 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:52:59,255 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:52:59,256 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/127.hdf5...
2023-02-03 11:52:59,805 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:00,218 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:00,219 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/128.hdf5...
2023-02-03 11:53:01,244 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:01,657 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:01,658 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/129.hdf5...
2023-02-03 11:53:02,280 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:02,692 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:02,693 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/130.hdf5...
2023-02-03 11:53:03,242 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:03,647 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:03,648 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/131.hdf5...
2023-02-03 11:53:04,270 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:04,679 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:04,680 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/132.hdf5...
2023-02-03 11:53:05,223 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:05,636 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:05,637 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/133.hdf5...
2023-02-03 11:53:06,190 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:06,593 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:06,594 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/134.hdf5...
2023-02-03 11:53:07,124 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:07,541 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:07,542 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/135.hdf5...
2023-02-03 11:53:08,307 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:08,708 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:08,709 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/136.hdf5...
2023-02-03 11:53:09,342 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:09,760 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:09,762 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/137.hdf5...
2023-02-03 11:53:10,356 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:10,771 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:10,772 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/138.hdf5...
2023-02-03 11:53:12,039 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:12,451 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:12,452 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/139.hdf5...
2023-02-03 11:53:13,061 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:13,481 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:13,482 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/140.hdf5...
2023-02-03 11:53:14,120 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:14,541 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:14,542 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/141.hdf5...
2023-02-03 11:53:15,182 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:15,591 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:15,592 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/142.hdf5...
2023-02-03 11:53:16,140 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:16,555 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:16,556 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/143.hdf5...
2023-02-03 11:53:17,207 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:17,625 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:17,626 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/144.hdf5...
2023-02-03 11:53:18,129 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:18,532 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:18,533 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/145.hdf5...
2023-02-03 11:53:19,145 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:19,560 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:19,561 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/146.hdf5...
2023-02-03 11:53:20,133 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:20,546 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:20,547 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/147.hdf5...
2023-02-03 11:53:21,104 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:21,518 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:21,519 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/148.hdf5...
2023-02-03 11:53:22,250 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:22,661 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:22,662 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/149.hdf5...
2023-02-03 11:53:23,999 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:24,404 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:24,404 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/150.hdf5...
2023-02-03 11:53:25,182 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:25,594 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:25,595 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/151.hdf5...
2023-02-03 11:53:26,413 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:26,832 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:26,833 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/152.hdf5...
2023-02-03 11:53:27,485 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:27,905 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:27,906 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/153.hdf5...
2023-02-03 11:53:28,481 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:28,892 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:28,893 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/154.hdf5...
2023-02-03 11:53:29,414 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:29,829 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:29,830 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/155.hdf5...
2023-02-03 11:53:30,351 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:30,765 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:30,766 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/156.hdf5...
2023-02-03 11:53:31,208 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:31,618 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:31,618 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/157.hdf5...
2023-02-03 11:53:32,174 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:32,582 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:32,583 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/158.hdf5...
2023-02-03 11:53:33,253 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:33,667 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:33,668 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/159.hdf5...
2023-02-03 11:53:34,442 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:34,858 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:34,859 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/160.hdf5...
2023-02-03 11:53:35,471 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:35,869 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:35,870 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/161.hdf5...
2023-02-03 11:53:36,310 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:36,712 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:36,713 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/162.hdf5...
2023-02-03 11:53:37,264 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:37,680 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:37,681 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/163.hdf5...
2023-02-03 11:53:38,135 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:38,543 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:38,544 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/164.hdf5...
2023-02-03 11:53:39,183 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:39,594 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:39,595 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/165.hdf5...
2023-02-03 11:53:40,170 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:40,582 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:40,583 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/166.hdf5...
2023-02-03 11:53:41,196 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:41,611 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:41,612 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/167.hdf5...
2023-02-03 11:53:42,410 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:42,815 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:42,816 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/168.hdf5...
2023-02-03 11:53:43,413 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:43,820 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:43,821 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/169.hdf5...
2023-02-03 11:53:44,455 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:44,872 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:44,873 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/170.hdf5...
2023-02-03 11:53:45,443 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:45,861 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:45,862 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/171.hdf5...
2023-02-03 11:53:46,320 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:46,728 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:46,729 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/172.hdf5...
2023-02-03 11:53:47,238 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:47,653 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:47,654 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/173.hdf5...
2023-02-03 11:53:48,267 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:48,665 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:48,666 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/174.hdf5...
2023-02-03 11:53:49,280 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:49,699 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:49,700 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/175.hdf5...
2023-02-03 11:53:50,107 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:50,522 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:50,523 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/176.hdf5...
2023-02-03 11:53:51,230 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:51,637 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:51,638 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/177.hdf5...
2023-02-03 11:53:52,351 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:52,768 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:52,769 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/178.hdf5...
2023-02-03 11:53:53,396 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:53,814 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:53,815 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/179.hdf5...
2023-02-03 11:53:54,257 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:54,675 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:54,676 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/180.hdf5...
2023-02-03 11:53:55,273 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:55,687 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:55,688 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/181.hdf5...
2023-02-03 11:53:56,299 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:56,707 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:56,708 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/182.hdf5...
2023-02-03 11:53:57,184 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:57,590 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:57,591 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/183.hdf5...
2023-02-03 11:53:58,081 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:58,496 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:58,497 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/184.hdf5...
2023-02-03 11:53:58,940 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:53:59,358 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:53:59,359 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/185.hdf5...
2023-02-03 11:54:00,008 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:00,415 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:00,416 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/186.hdf5...
2023-02-03 11:54:00,942 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:01,354 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:01,355 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/187.hdf5...
2023-02-03 11:54:01,801 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:02,217 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:02,218 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/188.hdf5...
2023-02-03 11:54:03,134 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:03,543 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:03,544 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/189.hdf5...
2023-02-03 11:54:04,071 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:04,487 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:04,488 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/190.hdf5...
2023-02-03 11:54:05,029 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:05,440 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:05,441 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/191.hdf5...
2023-02-03 11:54:06,200 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:06,610 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:06,611 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/192.hdf5...
2023-02-03 11:54:07,160 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:07,574 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:07,575 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/193.hdf5...
2023-02-03 11:54:08,264 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:08,677 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:08,678 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/194.hdf5...
2023-02-03 11:54:09,237 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:09,653 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:09,654 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/195.hdf5...
2023-02-03 11:54:10,430 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:10,850 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:10,851 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/196.hdf5...
2023-02-03 11:54:11,559 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:11,970 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:11,970 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/197.hdf5...
2023-02-03 11:54:12,675 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:13,077 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:13,078 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/198.hdf5...
2023-02-03 11:54:13,901 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:14,314 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:14,315 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/199.hdf5...
2023-02-03 11:54:14,980 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:15,394 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:15,395 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/200.hdf5...
2023-02-03 11:54:15,928 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:16,346 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:16,347 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/201.hdf5...
2023-02-03 11:54:16,863 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:17,274 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:17,275 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/202.hdf5...
2023-02-03 11:54:17,654 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:18,065 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:18,066 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/203.hdf5...
2023-02-03 11:54:18,648 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:19,056 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:19,057 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/204.hdf5...
2023-02-03 11:54:19,642 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:20,049 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:20,050 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/205.hdf5...
2023-02-03 11:54:20,759 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:21,164 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:21,165 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/206.hdf5...
2023-02-03 11:54:21,655 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:22,071 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:22,071 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/207.hdf5...
2023-02-03 11:54:22,745 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:23,156 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:23,157 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/208.hdf5...
2023-02-03 11:54:24,185 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:24,593 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:24,594 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/209.hdf5...
2023-02-03 11:54:25,808 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:26,216 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:26,217 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/210.hdf5...
2023-02-03 11:54:26,735 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:27,153 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:27,154 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/211.hdf5...
2023-02-03 11:54:27,574 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:27,987 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:27,988 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/212.hdf5...
2023-02-03 11:54:28,508 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:28,918 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:28,919 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/213.hdf5...
2023-02-03 11:54:29,483 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:29,901 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:29,902 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/214.hdf5...
2023-02-03 11:54:30,388 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:30,801 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:30,802 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/215.hdf5...
2023-02-03 11:54:31,462 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:31,877 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:31,878 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/216.hdf5...
2023-02-03 11:54:32,675 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:33,082 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:33,083 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/217.hdf5...
2023-02-03 11:54:33,733 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:34,141 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:34,142 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/218.hdf5...
2023-02-03 11:54:34,623 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:35,036 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:35,037 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/219.hdf5...
2023-02-03 11:54:35,431 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:35,848 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:35,849 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/220.hdf5...
2023-02-03 11:54:36,336 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:36,754 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:36,755 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/221.hdf5...
2023-02-03 11:54:37,317 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:37,732 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:37,733 [MainThread] INFO HDF5Dataset - Loading train set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/train/222.hdf5...
2023-02-03 11:54:38,295 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:38,697 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:38,701 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/0.hdf5...
2023-02-03 11:54:39,424 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:39,828 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:39,829 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/1.hdf5...
2023-02-03 11:54:40,501 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:40,919 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:40,919 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/2.hdf5...
2023-02-03 11:54:41,759 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:42,174 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:42,175 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/3.hdf5...
2023-02-03 11:54:42,639 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:43,055 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:43,056 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/4.hdf5...
2023-02-03 11:54:43,693 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:44,105 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:44,106 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/5.hdf5...
2023-02-03 11:54:44,710 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:45,117 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:45,118 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/6.hdf5...
2023-02-03 11:54:45,673 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:46,086 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:46,087 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/7.hdf5...
2023-02-03 11:54:46,818 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:47,227 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:47,228 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/8.hdf5...
2023-02-03 11:54:47,854 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:48,262 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:48,263 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/9.hdf5...
2023-02-03 11:54:48,753 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:49,151 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:49,152 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/10.hdf5...
2023-02-03 11:54:49,967 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:50,378 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:50,379 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/11.hdf5...
2023-02-03 11:54:50,785 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:51,193 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:51,194 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/12.hdf5...
2023-02-03 11:54:51,579 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:51,993 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:51,994 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/13.hdf5...
2023-02-03 11:54:52,462 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:52,874 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:52,875 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/14.hdf5...
2023-02-03 11:54:53,431 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:53,835 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:53,836 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/15.hdf5...
2023-02-03 11:54:54,208 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:54,618 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:54,619 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/16.hdf5...
2023-02-03 11:54:55,099 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:55,515 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:55,516 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/17.hdf5...
2023-02-03 11:54:56,129 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:56,537 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:56,538 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/18.hdf5...
2023-02-03 11:54:57,583 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:57,997 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:57,998 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/19.hdf5...
2023-02-03 11:54:58,551 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:58,959 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:58,960 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/20.hdf5...
2023-02-03 11:54:59,526 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:54:59,924 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:54:59,925 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/21.hdf5...
2023-02-03 11:55:00,459 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:00,876 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:00,877 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/22.hdf5...
2023-02-03 11:55:01,550 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:01,960 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:01,961 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/23.hdf5...
2023-02-03 11:55:02,609 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:03,020 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:03,020 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/24.hdf5...
2023-02-03 11:55:03,580 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:03,993 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:03,994 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/25.hdf5...
2023-02-03 11:55:04,445 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:04,861 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:04,862 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/26.hdf5...
2023-02-03 11:55:05,473 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:05,889 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:05,890 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/27.hdf5...
2023-02-03 11:55:06,497 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:06,909 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:06,910 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/28.hdf5...
2023-02-03 11:55:07,706 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:08,108 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:08,108 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/29.hdf5...
2023-02-03 11:55:08,570 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:08,978 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:08,979 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/30.hdf5...
2023-02-03 11:55:09,386 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:09,798 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:09,799 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/31.hdf5...
2023-02-03 11:55:10,229 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:10,638 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:10,639 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/32.hdf5...
2023-02-03 11:55:11,254 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:11,671 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:11,672 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/33.hdf5...
2023-02-03 11:55:12,436 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:12,846 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:12,847 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/34.hdf5...
2023-02-03 11:55:13,265 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:13,670 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:13,671 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/35.hdf5...
2023-02-03 11:55:14,245 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:14,658 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:14,659 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/36.hdf5...
2023-02-03 11:55:15,106 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:15,514 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:15,515 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/37.hdf5...
2023-02-03 11:55:16,421 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:16,836 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:16,837 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/38.hdf5...
2023-02-03 11:55:17,539 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:17,945 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:17,946 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/39.hdf5...
2023-02-03 11:55:18,305 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:18,711 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:18,712 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/40.hdf5...
2023-02-03 11:55:19,191 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:19,606 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:19,607 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/41.hdf5...
2023-02-03 11:55:20,301 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:20,711 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:20,712 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/42.hdf5...
2023-02-03 11:55:21,247 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:21,663 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:21,664 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/43.hdf5...
2023-02-03 11:55:22,229 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:22,635 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:22,636 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/44.hdf5...
2023-02-03 11:55:23,120 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:23,535 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:23,536 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/45.hdf5...
2023-02-03 11:55:23,922 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:24,338 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:24,339 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/46.hdf5...
2023-02-03 11:55:24,715 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:25,123 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:25,123 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/47.hdf5...
2023-02-03 11:55:25,539 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:25,951 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:25,952 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/48.hdf5...
2023-02-03 11:55:26,376 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:26,787 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:26,788 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/49.hdf5...
2023-02-03 11:55:27,328 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:27,727 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:27,728 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/50.hdf5...
2023-02-03 11:55:28,197 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:28,613 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:28,614 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/51.hdf5...
2023-02-03 11:55:29,069 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:29,475 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:29,476 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/52.hdf5...
2023-02-03 11:55:29,867 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:30,280 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:30,280 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/53.hdf5...
2023-02-03 11:55:31,584 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:31,991 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:31,992 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/54.hdf5...
2023-02-03 11:55:32,469 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:32,883 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:32,884 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/55.hdf5...
2023-02-03 11:55:33,730 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:34,137 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:34,138 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/56.hdf5...
2023-02-03 11:55:34,654 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:35,060 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:35,061 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/57.hdf5...
2023-02-03 11:55:35,588 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:36,004 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:36,005 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/58.hdf5...
2023-02-03 11:55:36,613 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:37,024 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:37,025 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/59.hdf5...
2023-02-03 11:55:37,519 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:37,935 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:37,936 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/60.hdf5...
2023-02-03 11:55:38,440 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:38,852 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:38,853 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/61.hdf5...
2023-02-03 11:55:39,584 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:39,981 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:39,982 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/62.hdf5...
2023-02-03 11:55:40,456 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:40,873 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:40,874 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/63.hdf5...
2023-02-03 11:55:41,393 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:41,805 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:41,805 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/64.hdf5...
2023-02-03 11:55:42,284 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:42,693 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:42,694 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/65.hdf5...
2023-02-03 11:55:43,157 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:43,572 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:43,573 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/66.hdf5...
2023-02-03 11:55:44,051 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:44,455 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:44,456 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/67.hdf5...
2023-02-03 11:55:44,878 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:45,288 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:45,289 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/68.hdf5...
2023-02-03 11:55:45,716 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:46,128 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:46,129 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/69.hdf5...
2023-02-03 11:55:46,571 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:46,981 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:46,982 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/70.hdf5...
2023-02-03 11:55:47,498 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:47,895 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:47,896 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/71.hdf5...
2023-02-03 11:55:48,303 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:48,719 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:48,720 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/72.hdf5...
2023-02-03 11:55:49,125 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:49,546 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:49,547 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/73.hdf5...
2023-02-03 11:55:50,138 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:50,548 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:50,549 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/74.hdf5...
2023-02-03 11:55:50,986 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:51,468 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:51,469 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/75.hdf5...
2023-02-03 11:55:52,036 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:52,443 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:52,444 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/76.hdf5...
2023-02-03 11:55:52,799 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:53,218 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:53,220 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/77.hdf5...
2023-02-03 11:55:53,607 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:54,018 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:54,019 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/78.hdf5...
2023-02-03 11:55:54,466 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:54,882 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:54,883 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/79.hdf5...
2023-02-03 11:55:55,347 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:55,757 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:55,758 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/80.hdf5...
2023-02-03 11:55:56,395 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:56,799 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:56,800 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/81.hdf5...
2023-02-03 11:55:57,348 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:57,756 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:57,757 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/82.hdf5...
2023-02-03 11:55:58,462 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:58,864 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:58,865 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/83.hdf5...
2023-02-03 11:55:59,461 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:55:59,876 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:55:59,877 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/84.hdf5...
2023-02-03 11:56:00,673 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:56:01,082 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:56:01,083 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/85.hdf5...
2023-02-03 11:56:01,726 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:56:02,135 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:56:02,136 [MainThread] INFO HDF5Dataset - Loading val set from: /projects/p_bioimage/laura/data/hdf5_for_plantseg/val/86.hdf5...
2023-02-03 11:56:02,700 [MainThread] INFO Dataset - Slice builder config: {'name': 'FilterSliceBuilder', 'patch_shape': [64, 260, 260], 'stride_shape': [1, 130, 130], 'threshold': 0.6, 'slack_acceptance': 0.01}
2023-02-03 11:56:03,106 [MainThread] INFO HDF5Dataset - Number of patches: 1
2023-02-03 11:56:03,107 [MainThread] INFO Dataset - Number of workers for train/val dataloader: 6
2023-02-03 11:56:03,107 [MainThread] INFO Dataset - 4 GPUs available. Using batch_size = 4 * 1
2023-02-03 11:56:03,108 [MainThread] INFO Dataset - Batch size for train/val loader: 4
2023-02-03 11:56:03,109 [MainThread] INFO UNet3DTrainer - DataParallel(
(module): UNet3D(
(encoders): ModuleList(
(0): Encoder(
(basic_module): DoubleConv(
(SingleConv1): SingleConv(
(groupnorm): GroupNorm(1, 1, eps=1e-05, affine=True)
(conv): Conv3d(1, 16, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
(SingleConv2): SingleConv(
(groupnorm): GroupNorm(8, 16, eps=1e-05, affine=True)
(conv): Conv3d(16, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
)
)
(1): Encoder(
(pooling): MaxPool3d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(basic_module): DoubleConv(
(SingleConv1): SingleConv(
(groupnorm): GroupNorm(8, 32, eps=1e-05, affine=True)
(conv): Conv3d(32, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
(SingleConv2): SingleConv(
(groupnorm): GroupNorm(8, 32, eps=1e-05, affine=True)
(conv): Conv3d(32, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
)
)
(2): Encoder(
(pooling): MaxPool3d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(basic_module): DoubleConv(
(SingleConv1): SingleConv(
(groupnorm): GroupNorm(8, 64, eps=1e-05, affine=True)
(conv): Conv3d(64, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
(SingleConv2): SingleConv(
(groupnorm): GroupNorm(8, 64, eps=1e-05, affine=True)
(conv): Conv3d(64, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
)
)
(3): Encoder(
(pooling): MaxPool3d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
(basic_module): DoubleConv(
(SingleConv1): SingleConv(
(groupnorm): GroupNorm(8, 128, eps=1e-05, affine=True)
(conv): Conv3d(128, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
(SingleConv2): SingleConv(
(groupnorm): GroupNorm(8, 128, eps=1e-05, affine=True)
(conv): Conv3d(128, 256, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
)
)
)
(decoders): ModuleList(
(0): Decoder(
(upsampling): InterpolateUpsampling()
(basic_module): DoubleConv(
(SingleConv1): SingleConv(
(groupnorm): GroupNorm(8, 384, eps=1e-05, affine=True)
(conv): Conv3d(384, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
(SingleConv2): SingleConv(
(groupnorm): GroupNorm(8, 128, eps=1e-05, affine=True)
(conv): Conv3d(128, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
)
)
(1): Decoder(
(upsampling): InterpolateUpsampling()
(basic_module): DoubleConv(
(SingleConv1): SingleConv(
(groupnorm): GroupNorm(8, 192, eps=1e-05, affine=True)
(conv): Conv3d(192, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
(SingleConv2): SingleConv(
(groupnorm): GroupNorm(8, 64, eps=1e-05, affine=True)
(conv): Conv3d(64, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
)
)
(2): Decoder(
(upsampling): InterpolateUpsampling()
(basic_module): DoubleConv(
(SingleConv1): SingleConv(
(groupnorm): GroupNorm(8, 96, eps=1e-05, affine=True)
(conv): Conv3d(96, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
(SingleConv2): SingleConv(
(groupnorm): GroupNorm(8, 32, eps=1e-05, affine=True)
(conv): Conv3d(32, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1), bias=False)
(ReLU): ReLU(inplace=True)
)
)
)
)
(final_conv): Conv3d(32, 1, kernel_size=(1, 1, 1), stride=(1, 1, 1))
(final_activation): Sigmoid()
)
)
2023-02-03 11:56:03,111 [MainThread] INFO UNet3DTrainer - eval_score_higher_is_better: False
2023-02-03 11:56:30,940 [MainThread] INFO UNet3DTrainer - Training iteration [1/60000]. Epoch [0/499]
2023-02-03 11:56:49,904 [MainThread] INFO UNet3DTrainer - Training iteration [2/60000]. Epoch [0/499]
2023-02-03 11:56:52,316 [MainThread] INFO UNet3DTrainer - Training iteration [3/60000]. Epoch [0/499]
2023-02-03 11:56:52,694 [MainThread] INFO UNet3DTrainer - Training iteration [4/60000]. Epoch [0/499]
2023-02-03 11:56:53,248 [MainThread] INFO UNet3DTrainer - Training iteration [5/60000]. Epoch [0/499]
2023-02-03 11:56:53,793 [MainThread] INFO UNet3DTrainer - Training iteration [6/60000]. Epoch [0/499]
2023-02-03 11:56:54,371 [MainThread] INFO UNet3DTrainer - Training iteration [7/60000]. Epoch [0/499]
2023-02-03 11:56:54,970 [MainThread] INFO UNet3DTrainer - Training iteration [8/60000]. Epoch [0/499]
2023-02-03 11:56:57,116 [MainThread] INFO UNet3DTrainer - Training iteration [9/60000]. Epoch [0/499]
2023-02-03 11:56:57,523 [MainThread] INFO UNet3DTrainer - Training iteration [10/60000]. Epoch [0/499]
2023-02-03 11:56:58,103 [MainThread] INFO UNet3DTrainer - Training iteration [11/60000]. Epoch [0/499]
2023-02-03 11:56:58,690 [MainThread] INFO UNet3DTrainer - Training iteration [12/60000]. Epoch [0/499]
2023-02-03 11:57:23,980 [MainThread] INFO UNet3DTrainer - Training iteration [13/60000]. Epoch [0/499]
2023-02-03 11:57:24,407 [MainThread] INFO UNet3DTrainer - Training iteration [14/60000]. Epoch [0/499]
2023-02-03 11:57:25,024 [MainThread] INFO UNet3DTrainer - Training iteration [15/60000]. Epoch [0/499]
2023-02-03 11:57:25,631 [MainThread] INFO UNet3DTrainer - Training iteration [16/60000]. Epoch [0/499]
2023-02-03 11:57:48,462 [MainThread] INFO UNet3DTrainer - Training iteration [17/60000]. Epoch [0/499]
2023-02-03 11:57:48,848 [MainThread] INFO UNet3DTrainer - Training iteration [18/60000]. Epoch [0/499]
2023-02-03 11:57:52,610 [MainThread] INFO UNet3DTrainer - Training iteration [19/60000]. Epoch [0/499]
2023-02-03 11:59:50,058 [MainThread] INFO UNet3DTrainer - Training iteration [51/60000]. Epoch [0/499]
2023-02-03 11:59:50,431 [MainThread] INFO UNet3DTrainer - Training iteration [52/60000]. Epoch [0/499]
2023-02-03 11:59:50,959 [MainThread] INFO UNet3DTrainer - Training iteration [53/60000]. Epoch [0/499]
2023-02-03 11:59:51,489 [MainThread] INFO UNet3DTrainer - Training iteration [54/60000]. Epoch [0/499]
2023-02-03 11:59:52,009 [MainThread] INFO UNet3DTrainer - Training iteration [55/60000]. Epoch [0/499]
2023-02-03 11:59:54,721 [MainThread] INFO UNet3DTrainer - Training iteration [56/60000]. Epoch [0/499]
2023-02-03 12:00:03,109 [MainThread] INFO UNet3DTrainer - Training iteration [57/60000]. Epoch [1/499]
2023-02-03 12:00:23,474 [MainThread] INFO UNet3DTrainer - Training iteration [58/60000]. Epoch [1/499]
2023-02-03 12:00:23,839 [MainThread] INFO UNet3DTrainer - Training iteration [59/60000]. Epoch [1/499]
2023-02-03 12:00:24,396 [MainThread] INFO UNet3DTrainer - Training iteration [60/60000]. Epoch [1/499]
2023-02-03 12:00:24,941 [MainThread] INFO UNet3DTrainer - Training iteration [61/60000]. Epoch [1/499]
2023-02-03 12:00:25,491 [MainThread] INFO UNet3DTrainer - Training iteration [62/60000]. Epoch [1/499]
2023-02-03 12:00:26,045 [MainThread] INFO UNet3DTrainer - Training iteration [63/60000]. Epoch [1/499]
2023-02-03 12:00:46,515 [MainThread] INFO UNet3DTrainer - Training iteration [64/60000]. Epoch [1/499]
2023-02-03 12:00:46,896 [MainThread] INFO UNet3DTrainer - Training iteration [65/60000]. Epoch [1/499]
2023-02-03 12:00:47,432 [MainThread] INFO UNet3DTrainer - Training iteration [66/60000]. Epoch [1/499]
2023-02-03 12:00:47,973 [MainThread] INFO UNet3DTrainer - Training iteration [67/60000]. Epoch [1/499]
2023-02-03 12:00:48,532 [MainThread] INFO UNet3DTrainer - Training iteration [68/60000]. Epoch [1/499]
2023-02-03 12:00:49,079 [MainThread] INFO UNet3DTrainer - Training iteration [69/60000]. Epoch [1/499]
2023-02-03 12:00:50,718 [MainThread] INFO UNet3DTrainer - Training iteration [70/60000]. Epoch [1/499]
2023-02-03 12:00:51,174 [MainThread] INFO UNet3DTrainer - Training iteration [71/60000]. Epoch [1/499]
2023-02-03 12:00:51,809 [MainThread] INFO UNet3DTrainer - Training iteration [72/60000]. Epoch [1/499]
2023-02-03 12:00:52,471 [MainThread] INFO UNet3DTrainer - Training iteration [73/60000]. Epoch [1/499]
2023-02-03 12:00:53,008 [MainThread] INFO UNet3DTrainer - Training iteration [74/60000]. Epoch [1/499]
2023-02-03 12:00:53,604 [MainThread] INFO UNet3DTrainer - Training iteration [75/60000]. Epoch [1/499]
2023-02-03 12:00:56,164 [MainThread] INFO UNet3DTrainer - Training iteration [76/60000]. Epoch [1/499]
2023-02-03 12:01:35,277 [MainThread] INFO UNet3DTrainer - Training iteration [77/60000]. Epoch [1/499]
2023-02-03 12:01:35,641 [MainThread] INFO UNet3DTrainer - Training iteration [78/60000]. Epoch [1/499]
2023-02-03 12:01:36,190 [MainThread] INFO UNet3DTrainer - Training iteration [79/60000]. Epoch [1/499]
2023-02-03 12:01:36,727 [MainThread] INFO UNet3DTrainer - Training iteration [80/60000]. Epoch [1/499]
2023-02-03 12:01:37,282 [MainThread] INFO UNet3DTrainer - Training iteration [81/60000]. Epoch [1/499]
2023-02-03 12:01:37,920 [MainThread] INFO UNet3DTrainer - Training iteration [82/60000]. Epoch [1/499]
2023-02-03 12:01:40,101 [MainThread] INFO UNet3DTrainer - Training iteration [83/60000]. Epoch [1/499]
2023-02-03 12:01:40,533 [MainThread] INFO UNet3DTrainer - Training iteration [84/60000]. Epoch [1/499]
2023-02-03 12:01:41,090 [MainThread] INFO UNet3DTrainer - Training iteration [85/60000]. Epoch [1/499]
2023-02-03 12:01:41,700 [MainThread] INFO UNet3DTrainer - Training iteration [86/60000]. Epoch [1/499]
2023-02-03 12:01:42,260 [MainThread] INFO UNet3DTrainer - Training iteration [87/60000]. Epoch [1/499]
2023-02-03 12:02:14,319 [MainThread] INFO UNet3DTrainer - Training iteration [88/60000]. Epoch [1/499]
2023-02-03 12:02:14,719 [MainThread] INFO UNet3DTrainer - Training iteration [89/60000]. Epoch [1/499]
2023-02-03 12:02:15,255 [MainThread] INFO UNet3DTrainer - Training iteration [90/60000]. Epoch [1/499]
2023-02-03 12:02:15,810 [MainThread] INFO UNet3DTrainer - Training iteration [91/60000]. Epoch [1/499]
2023-02-03 12:02:16,476 [MainThread] INFO UNet3DTrainer - Training iteration [92/60000]. Epoch [1/499]
2023-02-03 12:02:30,439 [MainThread] INFO UNet3DTrainer - Training iteration [93/60000]. Epoch [1/499]
2023-02-03 12:02:39,307 [MainThread] INFO UNet3DTrainer - Training iteration [94/60000]. Epoch [1/499]
2023-02-03 12:02:39,681 [MainThread] INFO UNet3DTrainer - Training iteration [95/60000]. Epoch [1/499]
2023-02-03 12:02:40,227 [MainThread] INFO UNet3DTrainer - Training iteration [96/60000]. Epoch [1/499]
2023-02-03 12:02:40,777 [MainThread] INFO UNet3DTrainer - Training iteration [97/60000]. Epoch [1/499]
2023-02-03 12:02:57,009 [MainThread] INFO UNet3DTrainer - Training iteration [98/60000]. Epoch [1/499]
2023-02-03 12:02:57,999 [MainThread] INFO UNet3DTrainer - Training iteration [99/60000]. Epoch [1/499]
2023-02-03 12:02:58,366 [MainThread] INFO UNet3DTrainer - Training iteration [100/60000]. Epoch [1/499]
2023-02-03 12:02:58,883 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 12:03:04,938 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 12:03:16,750 [MainThread] INFO EvalMetric - ARand: 0.6751594611156824
2023-02-03 12:03:16,798 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 12:03:25,218 [MainThread] INFO EvalMetric - ARand: 0.7849761260151421
2023-02-03 12:03:25,265 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 12:03:33,427 [MainThread] INFO EvalMetric - ARand: 0.8194119756151124
2023-02-03 12:03:33,469 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 12:03:41,471 [MainThread] INFO EvalMetric - ARand: 0.7346407355083011
2023-02-03 12:03:41,509 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 12:03:49,073 [MainThread] INFO EvalMetric - ARand: 0.7170494619353749
2023-02-03 12:03:49,118 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 12:03:56,702 [MainThread] INFO EvalMetric - ARand: 0.7812004605948055
2023-02-03 12:03:56,755 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 12:04:04,158 [MainThread] INFO EvalMetric - ARand: 0.6341318605224019
2023-02-03 12:04:04,201 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 12:04:11,522 [MainThread] INFO EvalMetric - ARand: 0.787845363054317
2023-02-03 12:04:11,567 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 12:04:18,971 [MainThread] INFO EvalMetric - ARand: 0.6944384465659982
2023-02-03 12:04:19,018 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 12:04:26,400 [MainThread] INFO EvalMetric - ARand: 0.746877210501596
2023-02-03 12:04:26,438 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 12:04:33,838 [MainThread] INFO EvalMetric - ARand: 0.6750768698500409
2023-02-03 12:04:33,879 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 12:04:41,279 [MainThread] INFO EvalMetric - ARand: 0.6638065847516527
2023-02-03 12:04:41,318 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 12:04:48,740 [MainThread] INFO EvalMetric - ARand: 0.7546460249998476
2023-02-03 12:04:48,783 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 12:04:56,153 [MainThread] INFO EvalMetric - ARand: 0.7695003365829094
2023-02-03 12:04:56,198 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 12:05:03,643 [MainThread] INFO EvalMetric - ARand: 0.6743625059987417
2023-02-03 12:05:03,691 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 12:05:11,215 [MainThread] INFO EvalMetric - ARand: 0.7422157337383821
2023-02-03 12:05:11,256 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 12:05:18,772 [MainThread] INFO EvalMetric - ARand: 0.7280543362984393
2023-02-03 12:05:18,816 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 12:05:26,287 [MainThread] INFO EvalMetric - ARand: 0.778076985712778
2023-02-03 12:05:26,328 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 12:05:33,781 [MainThread] INFO EvalMetric - ARand: 0.7946724876173114
2023-02-03 12:05:33,825 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 12:05:41,229 [MainThread] INFO EvalMetric - ARand: 0.7641986784411408
2023-02-03 12:05:41,272 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 12:05:48,626 [MainThread] INFO EvalMetric - ARand: 0.8285460393051802
2023-02-03 12:05:48,673 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 12:05:54,262 [MainThread] INFO EvalMetric - ARand: 0.8411439941969219
2023-02-03 12:05:54,722 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4941661357879639. Evaluation score: 0.7438963531205907
2023-02-03 12:05:54,749 [MainThread] INFO UNet3DTrainer - Saving new best evaluation metric: 0.7438963531205907
2023-02-03 12:05:54,751 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 12:06:02,239 [MainThread] INFO EvalMetric - ARand: 0.8195916215323724
2023-02-03 12:06:02,259 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.32454235038974066. Evaluation score: 0.8195916215323724
2023-02-03 12:06:02,260 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 12:06:03,319 [MainThread] INFO UNet3DTrainer - Training iteration [101/60000]. Epoch [1/499]
2023-02-03 12:06:03,676 [MainThread] INFO UNet3DTrainer - Training iteration [102/60000]. Epoch [1/499]
2023-02-03 12:06:04,203 [MainThread] INFO UNet3DTrainer - Training iteration [103/60000]. Epoch [1/499]
2023-02-03 12:06:04,737 [MainThread] INFO UNet3DTrainer - Training iteration [104/60000]. Epoch [1/499]
2023-02-03 12:06:05,268 [MainThread] INFO UNet3DTrainer - Training iteration [105/60000]. Epoch [1/499]
2023-02-03 12:06:05,792 [MainThread] INFO UNet3DTrainer - Training iteration [106/60000]. Epoch [1/499]
2023-02-03 12:06:06,322 [MainThread] INFO UNet3DTrainer - Training iteration [107/60000]. Epoch [1/499]
2023-02-03 12:06:06,841 [MainThread] INFO UNet3DTrainer - Training iteration [108/60000]. Epoch [1/499]
2023-02-03 12:06:07,372 [MainThread] INFO UNet3DTrainer - Training iteration [109/60000]. Epoch [1/499]
2023-02-03 12:06:07,903 [MainThread] INFO UNet3DTrainer - Training iteration [110/60000]. Epoch [1/499]
2023-02-03 12:06:08,429 [MainThread] INFO UNet3DTrainer - Training iteration [111/60000]. Epoch [1/499]
2023-02-03 12:06:08,974 [MainThread] INFO UNet3DTrainer - Training iteration [112/60000]. Epoch [1/499]
2023-02-03 12:06:42,264 [MainThread] INFO UNet3DTrainer - Training iteration [113/60000]. Epoch [2/499]
2023-02-03 12:06:42,750 [MainThread] INFO UNet3DTrainer - Training iteration [114/60000]. Epoch [2/499]
2023-02-03 12:06:43,403 [MainThread] INFO UNet3DTrainer - Training iteration [115/60000]. Epoch [2/499]
2023-02-03 12:06:43,960 [MainThread] INFO UNet3DTrainer - Training iteration [116/60000]. Epoch [2/499]
2023-02-03 12:06:44,520 [MainThread] INFO UNet3DTrainer - Training iteration [117/60000]. Epoch [2/499]
2023-02-03 12:06:45,090 [MainThread] INFO UNet3DTrainer - Training iteration [118/60000]. Epoch [2/499]
2023-02-03 12:07:08,567 [MainThread] INFO UNet3DTrainer - Training iteration [119/60000]. Epoch [2/499]
2023-02-03 12:07:11,282 [MainThread] INFO UNet3DTrainer - Training iteration [120/60000]. Epoch [2/499]
2023-02-03 12:07:11,668 [MainThread] INFO UNet3DTrainer - Training iteration [121/60000]. Epoch [2/499]
2023-02-03 12:07:12,215 [MainThread] INFO UNet3DTrainer - Training iteration [122/60000]. Epoch [2/499]
2023-02-03 12:07:30,681 [MainThread] INFO UNet3DTrainer - Training iteration [123/60000]. Epoch [2/499]
2023-02-03 12:07:31,079 [MainThread] INFO UNet3DTrainer - Training iteration [124/60000]. Epoch [2/499]
2023-02-03 12:07:34,234 [MainThread] INFO UNet3DTrainer - Training iteration [125/60000]. Epoch [2/499]
2023-02-03 12:07:36,803 [MainThread] INFO UNet3DTrainer - Training iteration [126/60000]. Epoch [2/499]
2023-02-03 12:07:37,212 [MainThread] INFO UNet3DTrainer - Training iteration [127/60000]. Epoch [2/499]
2023-02-03 12:07:37,747 [MainThread] INFO UNet3DTrainer - Training iteration [128/60000]. Epoch [2/499]
2023-02-03 12:08:15,983 [MainThread] INFO UNet3DTrainer - Training iteration [129/60000]. Epoch [2/499]
2023-02-03 12:08:16,358 [MainThread] INFO UNet3DTrainer - Training iteration [130/60000]. Epoch [2/499]
2023-02-03 12:08:19,447 [MainThread] INFO UNet3DTrainer - Training iteration [131/60000]. Epoch [2/499]
2023-02-03 12:08:19,820 [MainThread] INFO UNet3DTrainer - Training iteration [132/60000]. Epoch [2/499]
2023-02-03 12:08:20,357 [MainThread] INFO UNet3DTrainer - Training iteration [133/60000]. Epoch [2/499]
2023-02-03 12:08:20,902 [MainThread] INFO UNet3DTrainer - Training iteration [134/60000]. Epoch [2/499]
2023-02-03 12:08:37,911 [MainThread] INFO UNet3DTrainer - Training iteration [135/60000]. Epoch [2/499]
2023-02-03 12:08:38,269 [MainThread] INFO UNet3DTrainer - Training iteration [136/60000]. Epoch [2/499]
2023-02-03 12:08:38,797 [MainThread] INFO UNet3DTrainer - Training iteration [137/60000]. Epoch [2/499]
2023-02-03 12:08:39,345 [MainThread] INFO UNet3DTrainer - Training iteration [138/60000]. Epoch [2/499]
2023-02-03 12:08:39,875 [MainThread] INFO UNet3DTrainer - Training iteration [139/60000]. Epoch [2/499]
2023-02-03 12:08:40,447 [MainThread] INFO UNet3DTrainer - Training iteration [140/60000]. Epoch [2/499]
2023-02-03 12:08:41,907 [MainThread] INFO UNet3DTrainer - Training iteration [141/60000]. Epoch [2/499]
2023-02-03 12:08:42,293 [MainThread] INFO UNet3DTrainer - Training iteration [142/60000]. Epoch [2/499]
2023-02-03 12:08:47,553 [MainThread] INFO UNet3DTrainer - Training iteration [143/60000]. Epoch [2/499]
2023-02-03 12:08:47,935 [MainThread] INFO UNet3DTrainer - Training iteration [144/60000]. Epoch [2/499]
2023-02-03 12:08:48,485 [MainThread] INFO UNet3DTrainer - Training iteration [145/60000]. Epoch [2/499]
2023-02-03 12:08:49,035 [MainThread] INFO UNet3DTrainer - Training iteration [146/60000]. Epoch [2/499]
2023-02-03 12:08:49,583 [MainThread] INFO UNet3DTrainer - Training iteration [147/60000]. Epoch [2/499]
2023-02-03 12:08:50,147 [MainThread] INFO UNet3DTrainer - Training iteration [148/60000]. Epoch [2/499]
2023-02-03 12:08:52,181 [MainThread] INFO UNet3DTrainer - Training iteration [149/60000]. Epoch [2/499]
2023-02-03 12:09:06,687 [MainThread] INFO UNet3DTrainer - Training iteration [150/60000]. Epoch [2/499]
2023-02-03 12:09:07,075 [MainThread] INFO UNet3DTrainer - Training iteration [151/60000]. Epoch [2/499]
2023-02-03 12:09:27,809 [MainThread] INFO UNet3DTrainer - Training iteration [152/60000]. Epoch [2/499]
2023-02-03 12:09:28,194 [MainThread] INFO UNet3DTrainer - Training iteration [153/60000]. Epoch [2/499]
2023-02-03 12:09:28,729 [MainThread] INFO UNet3DTrainer - Training iteration [154/60000]. Epoch [2/499]
2023-02-03 12:09:29,269 [MainThread] INFO UNet3DTrainer - Training iteration [155/60000]. Epoch [2/499]
2023-02-03 12:09:29,848 [MainThread] INFO UNet3DTrainer - Training iteration [156/60000]. Epoch [2/499]
2023-02-03 12:09:30,393 [MainThread] INFO UNet3DTrainer - Training iteration [157/60000]. Epoch [2/499]
2023-02-03 12:09:53,237 [MainThread] INFO UNet3DTrainer - Training iteration [158/60000]. Epoch [2/499]
2023-02-03 12:09:53,623 [MainThread] INFO UNet3DTrainer - Training iteration [159/60000]. Epoch [2/499]
2023-02-03 12:09:54,177 [MainThread] INFO UNet3DTrainer - Training iteration [160/60000]. Epoch [2/499]
2023-02-03 12:09:54,744 [MainThread] INFO UNet3DTrainer - Training iteration [161/60000]. Epoch [2/499]
2023-02-03 12:09:55,275 [MainThread] INFO UNet3DTrainer - Training iteration [162/60000]. Epoch [2/499]
2023-02-03 12:09:55,828 [MainThread] INFO UNet3DTrainer - Training iteration [163/60000]. Epoch [2/499]
2023-02-03 12:09:57,091 [MainThread] INFO UNet3DTrainer - Training iteration [164/60000]. Epoch [2/499]
2023-02-03 12:10:01,740 [MainThread] INFO UNet3DTrainer - Training iteration [165/60000]. Epoch [2/499]
2023-02-03 12:10:02,123 [MainThread] INFO UNet3DTrainer - Training iteration [166/60000]. Epoch [2/499]
2023-02-03 12:10:02,647 [MainThread] INFO UNet3DTrainer - Training iteration [167/60000]. Epoch [2/499]
2023-02-03 12:10:03,171 [MainThread] INFO UNet3DTrainer - Training iteration [168/60000]. Epoch [2/499]
2023-02-03 12:10:36,393 [MainThread] INFO UNet3DTrainer - Training iteration [169/60000]. Epoch [3/499]
2023-02-03 12:10:36,888 [MainThread] INFO UNet3DTrainer - Training iteration [170/60000]. Epoch [3/499]
2023-02-03 12:10:57,045 [MainThread] INFO UNet3DTrainer - Training iteration [171/60000]. Epoch [3/499]
2023-02-03 12:10:57,427 [MainThread] INFO UNet3DTrainer - Training iteration [172/60000]. Epoch [3/499]
2023-02-03 12:10:57,983 [MainThread] INFO UNet3DTrainer - Training iteration [173/60000]. Epoch [3/499]
2023-02-03 12:10:58,529 [MainThread] INFO UNet3DTrainer - Training iteration [174/60000]. Epoch [3/499]
2023-02-03 12:10:59,097 [MainThread] INFO UNet3DTrainer - Training iteration [175/60000]. Epoch [3/499]
2023-02-03 12:10:59,641 [MainThread] INFO UNet3DTrainer - Training iteration [176/60000]. Epoch [3/499]
2023-02-03 12:11:01,100 [MainThread] INFO UNet3DTrainer - Training iteration [177/60000]. Epoch [3/499]
2023-02-03 12:11:01,540 [MainThread] INFO UNet3DTrainer - Training iteration [178/60000]. Epoch [3/499]
2023-02-03 12:11:02,165 [MainThread] INFO UNet3DTrainer - Training iteration [179/60000]. Epoch [3/499]
2023-02-03 12:11:02,760 [MainThread] INFO UNet3DTrainer - Training iteration [180/60000]. Epoch [3/499]
2023-02-03 12:11:03,299 [MainThread] INFO UNet3DTrainer - Training iteration [181/60000]. Epoch [3/499]
2023-02-03 12:11:03,965 [MainThread] INFO UNet3DTrainer - Training iteration [182/60000]. Epoch [3/499]
2023-02-03 12:11:06,339 [MainThread] INFO UNet3DTrainer - Training iteration [183/60000]. Epoch [3/499]
2023-02-03 12:11:06,779 [MainThread] INFO UNet3DTrainer - Training iteration [184/60000]. Epoch [3/499]
2023-02-03 12:11:30,942 [MainThread] INFO UNet3DTrainer - Training iteration [185/60000]. Epoch [3/499]
2023-02-03 12:11:31,347 [MainThread] INFO UNet3DTrainer - Training iteration [186/60000]. Epoch [3/499]
2023-02-03 12:11:31,905 [MainThread] INFO UNet3DTrainer - Training iteration [187/60000]. Epoch [3/499]
2023-02-03 12:11:48,178 [MainThread] INFO UNet3DTrainer - Training iteration [188/60000]. Epoch [3/499]
2023-02-03 12:11:48,550 [MainThread] INFO UNet3DTrainer - Training iteration [189/60000]. Epoch [3/499]
2023-02-03 12:11:49,085 [MainThread] INFO UNet3DTrainer - Training iteration [190/60000]. Epoch [3/499]
2023-02-03 12:11:49,630 [MainThread] INFO UNet3DTrainer - Training iteration [191/60000]. Epoch [3/499]
2023-02-03 12:11:50,202 [MainThread] INFO UNet3DTrainer - Training iteration [192/60000]. Epoch [3/499]
2023-02-03 12:11:50,780 [MainThread] INFO UNet3DTrainer - Training iteration [193/60000]. Epoch [3/499]
2023-02-03 12:11:52,400 [MainThread] INFO UNet3DTrainer - Training iteration [194/60000]. Epoch [3/499]
2023-02-03 12:11:52,792 [MainThread] INFO UNet3DTrainer - Training iteration [195/60000]. Epoch [3/499]
2023-02-03 12:11:53,378 [MainThread] INFO UNet3DTrainer - Training iteration [196/60000]. Epoch [3/499]
2023-02-03 12:11:54,020 [MainThread] INFO UNet3DTrainer - Training iteration [197/60000]. Epoch [3/499]
2023-02-03 12:11:54,670 [MainThread] INFO UNet3DTrainer - Training iteration [198/60000]. Epoch [3/499]
2023-02-03 12:12:22,214 [MainThread] INFO UNet3DTrainer - Training iteration [199/60000]. Epoch [3/499]
2023-02-03 12:12:22,613 [MainThread] INFO UNet3DTrainer - Training iteration [200/60000]. Epoch [3/499]
2023-02-03 12:12:23,141 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 12:12:28,962 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 12:12:39,726 [MainThread] INFO EvalMetric - ARand: 0.6641215815615831
2023-02-03 12:12:39,774 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 12:12:47,851 [MainThread] INFO EvalMetric - ARand: 0.7896559353148056
2023-02-03 12:12:47,921 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 12:12:55,532 [MainThread] INFO EvalMetric - ARand: 0.8070073234338423
2023-02-03 12:12:55,571 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 12:13:02,885 [MainThread] INFO EvalMetric - ARand: 0.7231205264697973
2023-02-03 12:13:02,925 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 12:13:10,360 [MainThread] INFO EvalMetric - ARand: 0.7130272908394126
2023-02-03 12:13:10,401 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 12:13:17,821 [MainThread] INFO EvalMetric - ARand: 0.772935427765753
2023-02-03 12:13:17,862 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 12:13:25,362 [MainThread] INFO EvalMetric - ARand: 0.6715272073461289
2023-02-03 12:13:25,400 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 12:13:32,806 [MainThread] INFO EvalMetric - ARand: 0.750214748851239
2023-02-03 12:13:32,851 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 12:13:40,208 [MainThread] INFO EvalMetric - ARand: 0.727296116404436
2023-02-03 12:13:40,246 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 12:13:47,599 [MainThread] INFO EvalMetric - ARand: 0.7323828540188794
2023-02-03 12:13:47,636 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 12:13:55,167 [MainThread] INFO EvalMetric - ARand: 0.6770568145474085
2023-02-03 12:13:55,207 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 12:14:02,653 [MainThread] INFO EvalMetric - ARand: 0.6614076291280473
2023-02-03 12:14:02,693 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 12:14:10,090 [MainThread] INFO EvalMetric - ARand: 0.7428624851838659
2023-02-03 12:14:10,131 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 12:14:17,480 [MainThread] INFO EvalMetric - ARand: 0.7502230329919819
2023-02-03 12:14:17,517 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 12:14:24,915 [MainThread] INFO EvalMetric - ARand: 0.7105802572346069
2023-02-03 12:14:24,955 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 12:14:32,351 [MainThread] INFO EvalMetric - ARand: 0.7027186679701006
2023-02-03 12:14:32,391 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 12:14:39,969 [MainThread] INFO EvalMetric - ARand: 0.7112829347417664
2023-02-03 12:14:40,011 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 12:14:47,397 [MainThread] INFO EvalMetric - ARand: 0.7694290392376548
2023-02-03 12:14:47,441 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 12:14:54,812 [MainThread] INFO EvalMetric - ARand: 0.7576320503788118
2023-02-03 12:14:54,860 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 12:15:02,285 [MainThread] INFO EvalMetric - ARand: 0.7777787373352334
2023-02-03 12:15:02,328 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 12:15:09,760 [MainThread] INFO EvalMetric - ARand: 0.8306743167052654
2023-02-03 12:15:09,803 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 12:15:15,407 [MainThread] INFO EvalMetric - ARand: 0.7868674919184593
2023-02-03 12:15:15,802 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4883483250935872. Evaluation score: 0.7371533607539983
2023-02-03 12:15:15,826 [MainThread] INFO UNet3DTrainer - Saving new best evaluation metric: 0.7371533607539983
2023-02-03 12:15:15,827 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 12:15:23,357 [MainThread] INFO EvalMetric - ARand: 0.8548960866901038
2023-02-03 12:15:23,376 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.2715526749379933. Evaluation score: 0.8548960866901038
2023-02-03 12:15:23,377 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 12:15:24,450 [MainThread] INFO UNet3DTrainer - Training iteration [201/60000]. Epoch [3/499]
2023-02-03 12:15:24,817 [MainThread] INFO UNet3DTrainer - Training iteration [202/60000]. Epoch [3/499]
2023-02-03 12:15:25,356 [MainThread] INFO UNet3DTrainer - Training iteration [203/60000]. Epoch [3/499]
2023-02-03 12:15:25,898 [MainThread] INFO UNet3DTrainer - Training iteration [204/60000]. Epoch [3/499]
2023-02-03 12:15:26,445 [MainThread] INFO UNet3DTrainer - Training iteration [205/60000]. Epoch [3/499]
2023-02-03 12:15:26,990 [MainThread] INFO UNet3DTrainer - Training iteration [206/60000]. Epoch [3/499]
2023-02-03 12:15:27,545 [MainThread] INFO UNet3DTrainer - Training iteration [207/60000]. Epoch [3/499]
2023-02-03 12:15:28,140 [MainThread] INFO UNet3DTrainer - Training iteration [208/60000]. Epoch [3/499]
2023-02-03 12:15:28,766 [MainThread] INFO UNet3DTrainer - Training iteration [209/60000]. Epoch [3/499]
2023-02-03 12:15:29,360 [MainThread] INFO UNet3DTrainer - Training iteration [210/60000]. Epoch [3/499]
2023-02-03 12:15:29,961 [MainThread] INFO UNet3DTrainer - Training iteration [211/60000]. Epoch [3/499]
2023-02-03 12:15:30,540 [MainThread] INFO UNet3DTrainer - Training iteration [212/60000]. Epoch [3/499]
2023-02-03 12:15:31,114 [MainThread] INFO UNet3DTrainer - Training iteration [213/60000]. Epoch [3/499]
2023-02-03 12:15:31,663 [MainThread] INFO UNet3DTrainer - Training iteration [214/60000]. Epoch [3/499]
2023-02-03 12:15:56,195 [MainThread] INFO UNet3DTrainer - Training iteration [215/60000]. Epoch [3/499]
2023-02-03 12:15:56,623 [MainThread] INFO UNet3DTrainer - Training iteration [216/60000]. Epoch [3/499]
2023-02-03 12:15:57,250 [MainThread] INFO UNet3DTrainer - Training iteration [217/60000]. Epoch [3/499]
2023-02-03 12:15:57,849 [MainThread] INFO UNet3DTrainer - Training iteration [218/60000]. Epoch [3/499]
2023-02-03 12:16:18,369 [MainThread] INFO UNet3DTrainer - Training iteration [219/60000]. Epoch [3/499]
2023-02-03 12:16:18,749 [MainThread] INFO UNet3DTrainer - Training iteration [220/60000]. Epoch [3/499]
2023-02-03 12:16:19,287 [MainThread] INFO UNet3DTrainer - Training iteration [221/60000]. Epoch [3/499]
2023-02-03 12:16:19,816 [MainThread] INFO UNet3DTrainer - Training iteration [222/60000]. Epoch [3/499]
2023-02-03 12:16:20,345 [MainThread] INFO UNet3DTrainer - Training iteration [223/60000]. Epoch [3/499]
2023-02-03 12:16:20,873 [MainThread] INFO UNet3DTrainer - Training iteration [224/60000]. Epoch [3/499]
2023-02-03 12:16:28,752 [MainThread] INFO UNet3DTrainer - Training iteration [225/60000]. Epoch [4/499]
2023-02-03 12:16:29,450 [MainThread] INFO UNet3DTrainer - Training iteration [226/60000]. Epoch [4/499]
2023-02-03 12:16:54,397 [MainThread] INFO UNet3DTrainer - Training iteration [227/60000]. Epoch [4/499]
2023-02-03 12:16:54,804 [MainThread] INFO UNet3DTrainer - Training iteration [228/60000]. Epoch [4/499]
2023-02-03 12:16:55,401 [MainThread] INFO UNet3DTrainer - Training iteration [229/60000]. Epoch [4/499]
2023-02-03 12:17:16,613 [MainThread] INFO UNet3DTrainer - Training iteration [230/60000]. Epoch [4/499]
2023-02-03 12:17:16,999 [MainThread] INFO UNet3DTrainer - Training iteration [231/60000]. Epoch [4/499]
2023-02-03 12:17:17,553 [MainThread] INFO UNet3DTrainer - Training iteration [232/60000]. Epoch [4/499]
2023-02-03 12:17:22,054 [MainThread] INFO UNet3DTrainer - Training iteration [233/60000]. Epoch [4/499]
2023-02-03 12:17:22,466 [MainThread] INFO UNet3DTrainer - Training iteration [234/60000]. Epoch [4/499]
2023-02-03 12:17:23,016 [MainThread] INFO UNet3DTrainer - Training iteration [235/60000]. Epoch [4/499]
2023-02-03 12:17:43,739 [MainThread] INFO UNet3DTrainer - Training iteration [236/60000]. Epoch [4/499]
2023-02-03 12:17:44,136 [MainThread] INFO UNet3DTrainer - Training iteration [237/60000]. Epoch [4/499]
2023-02-03 12:17:44,684 [MainThread] INFO UNet3DTrainer - Training iteration [238/60000]. Epoch [4/499]
2023-02-03 12:17:45,253 [MainThread] INFO UNet3DTrainer - Training iteration [239/60000]. Epoch [4/499]
2023-02-03 12:17:45,830 [MainThread] INFO UNet3DTrainer - Training iteration [240/60000]. Epoch [4/499]
2023-02-03 12:17:46,400 [MainThread] INFO UNet3DTrainer - Training iteration [241/60000]. Epoch [4/499]
2023-02-03 12:18:12,985 [MainThread] INFO UNet3DTrainer - Training iteration [242/60000]. Epoch [4/499]
2023-02-03 12:18:13,389 [MainThread] INFO UNet3DTrainer - Training iteration [243/60000]. Epoch [4/499]
2023-02-03 12:18:13,940 [MainThread] INFO UNet3DTrainer - Training iteration [244/60000]. Epoch [4/499]
2023-02-03 12:18:14,505 [MainThread] INFO UNet3DTrainer - Training iteration [245/60000]. Epoch [4/499]
2023-02-03 12:18:15,048 [MainThread] INFO UNet3DTrainer - Training iteration [246/60000]. Epoch [4/499]
2023-02-03 12:18:15,709 [MainThread] INFO UNet3DTrainer - Training iteration [247/60000]. Epoch [4/499]
2023-02-03 12:18:41,627 [MainThread] INFO UNet3DTrainer - Training iteration [248/60000]. Epoch [4/499]
2023-02-03 12:18:42,010 [MainThread] INFO UNet3DTrainer - Training iteration [249/60000]. Epoch [4/499]
2023-02-03 12:18:42,571 [MainThread] INFO UNet3DTrainer - Training iteration [250/60000]. Epoch [4/499]
2023-02-03 12:18:43,133 [MainThread] INFO UNet3DTrainer - Training iteration [251/60000]. Epoch [4/499]
2023-02-03 12:18:43,687 [MainThread] INFO UNet3DTrainer - Training iteration [252/60000]. Epoch [4/499]
2023-02-03 12:18:44,234 [MainThread] INFO UNet3DTrainer - Training iteration [253/60000]. Epoch [4/499]
2023-02-03 12:18:46,403 [MainThread] INFO UNet3DTrainer - Training iteration [254/60000]. Epoch [4/499]
2023-02-03 12:18:46,814 [MainThread] INFO UNet3DTrainer - Training iteration [255/60000]. Epoch [4/499]
2023-02-03 12:19:03,093 [MainThread] INFO UNet3DTrainer - Training iteration [256/60000]. Epoch [4/499]
2023-02-03 12:19:03,479 [MainThread] INFO UNet3DTrainer - Training iteration [257/60000]. Epoch [4/499]
2023-02-03 12:19:04,030 [MainThread] INFO UNet3DTrainer - Training iteration [258/60000]. Epoch [4/499]
2023-02-03 12:19:04,570 [MainThread] INFO UNet3DTrainer - Training iteration [259/60000]. Epoch [4/499]
2023-02-03 12:19:05,149 [MainThread] INFO UNet3DTrainer - Training iteration [260/60000]. Epoch [4/499]
2023-02-03 12:19:08,102 [MainThread] INFO UNet3DTrainer - Training iteration [261/60000]. Epoch [4/499]
2023-02-03 12:19:08,523 [MainThread] INFO UNet3DTrainer - Training iteration [262/60000]. Epoch [4/499]
2023-02-03 12:19:09,078 [MainThread] INFO UNet3DTrainer - Training iteration [263/60000]. Epoch [4/499]
2023-02-03 12:19:09,921 [MainThread] INFO UNet3DTrainer - Training iteration [264/60000]. Epoch [4/499]
2023-02-03 12:19:10,318 [MainThread] INFO UNet3DTrainer - Training iteration [265/60000]. Epoch [4/499]
2023-02-03 12:19:10,912 [MainThread] INFO UNet3DTrainer - Training iteration [266/60000]. Epoch [4/499]
2023-02-03 12:19:12,941 [MainThread] INFO UNet3DTrainer - Training iteration [267/60000]. Epoch [4/499]
2023-02-03 12:19:35,043 [MainThread] INFO UNet3DTrainer - Training iteration [268/60000]. Epoch [4/499]
2023-02-03 12:19:35,451 [MainThread] INFO UNet3DTrainer - Training iteration [269/60000]. Epoch [4/499]
2023-02-03 12:19:36,006 [MainThread] INFO UNet3DTrainer - Training iteration [270/60000]. Epoch [4/499]
2023-02-03 12:19:36,567 [MainThread] INFO UNet3DTrainer - Training iteration [271/60000]. Epoch [4/499]
2023-02-03 12:19:37,124 [MainThread] INFO UNet3DTrainer - Training iteration [272/60000]. Epoch [4/499]
2023-02-03 12:19:37,695 [MainThread] INFO UNet3DTrainer - Training iteration [273/60000]. Epoch [4/499]
2023-02-03 12:19:39,228 [MainThread] INFO UNet3DTrainer - Training iteration [274/60000]. Epoch [4/499]
2023-02-03 12:19:39,583 [MainThread] INFO UNet3DTrainer - Training iteration [275/60000]. Epoch [4/499]
2023-02-03 12:19:41,822 [MainThread] INFO UNet3DTrainer - Training iteration [276/60000]. Epoch [4/499]
2023-02-03 12:19:42,152 [MainThread] INFO UNet3DTrainer - Training iteration [277/60000]. Epoch [4/499]
2023-02-03 12:19:42,680 [MainThread] INFO UNet3DTrainer - Training iteration [278/60000]. Epoch [4/499]
2023-02-03 12:20:01,705 [MainThread] INFO UNet3DTrainer - Training iteration [279/60000]. Epoch [4/499]
2023-02-03 12:20:02,071 [MainThread] INFO UNet3DTrainer - Training iteration [280/60000]. Epoch [4/499]
2023-02-03 12:20:31,385 [MainThread] INFO UNet3DTrainer - Training iteration [281/60000]. Epoch [5/499]
2023-02-03 12:20:31,782 [MainThread] INFO UNet3DTrainer - Training iteration [282/60000]. Epoch [5/499]
2023-02-03 12:20:32,326 [MainThread] INFO UNet3DTrainer - Training iteration [283/60000]. Epoch [5/499]
2023-02-03 12:20:32,878 [MainThread] INFO UNet3DTrainer - Training iteration [284/60000]. Epoch [5/499]
2023-02-03 12:20:33,435 [MainThread] INFO UNet3DTrainer - Training iteration [285/60000]. Epoch [5/499]
2023-02-03 12:20:34,050 [MainThread] INFO UNet3DTrainer - Training iteration [286/60000]. Epoch [5/499]
2023-02-03 12:20:59,177 [MainThread] INFO UNet3DTrainer - Training iteration [287/60000]. Epoch [5/499]
2023-02-03 12:20:59,576 [MainThread] INFO UNet3DTrainer - Training iteration [288/60000]. Epoch [5/499]
2023-02-03 12:21:00,130 [MainThread] INFO UNet3DTrainer - Training iteration [289/60000]. Epoch [5/499]
2023-02-03 12:21:00,681 [MainThread] INFO UNet3DTrainer - Training iteration [290/60000]. Epoch [5/499]
2023-02-03 12:21:01,269 [MainThread] INFO UNet3DTrainer - Training iteration [291/60000]. Epoch [5/499]
2023-02-03 12:21:01,902 [MainThread] INFO UNet3DTrainer - Training iteration [292/60000]. Epoch [5/499]
2023-02-03 12:21:26,283 [MainThread] INFO UNet3DTrainer - Training iteration [293/60000]. Epoch [5/499]
2023-02-03 12:21:26,670 [MainThread] INFO UNet3DTrainer - Training iteration [294/60000]. Epoch [5/499]
2023-02-03 12:21:27,227 [MainThread] INFO UNet3DTrainer - Training iteration [295/60000]. Epoch [5/499]
2023-02-03 12:21:27,790 [MainThread] INFO UNet3DTrainer - Training iteration [296/60000]. Epoch [5/499]
2023-02-03 12:21:28,347 [MainThread] INFO UNet3DTrainer - Training iteration [297/60000]. Epoch [5/499]
2023-02-03 12:21:29,812 [MainThread] INFO UNet3DTrainer - Training iteration [298/60000]. Epoch [5/499]
2023-02-03 12:21:30,920 [MainThread] INFO UNet3DTrainer - Training iteration [299/60000]. Epoch [5/499]
2023-02-03 12:21:31,385 [MainThread] INFO UNet3DTrainer - Training iteration [300/60000]. Epoch [5/499]
2023-02-03 12:21:32,016 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 12:21:38,764 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 12:21:52,348 [MainThread] INFO EvalMetric - ARand: 0.6806544089322017
2023-02-03 12:21:52,392 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 12:22:04,323 [MainThread] INFO EvalMetric - ARand: 0.7732834828210238
2023-02-03 12:22:04,377 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 12:22:14,349 [MainThread] INFO EvalMetric - ARand: 0.8058246918382226
2023-02-03 12:22:14,405 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 12:22:22,426 [MainThread] INFO EvalMetric - ARand: 0.6962839277835816
2023-02-03 12:22:22,478 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 12:22:30,593 [MainThread] INFO EvalMetric - ARand: 0.7288378558886457
2023-02-03 12:22:30,642 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 12:22:38,197 [MainThread] INFO EvalMetric - ARand: 0.7698945447566544
2023-02-03 12:22:38,250 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 12:22:45,693 [MainThread] INFO EvalMetric - ARand: 0.7011419820752903
2023-02-03 12:22:45,730 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 12:22:53,040 [MainThread] INFO EvalMetric - ARand: 0.7530812908909497
2023-02-03 12:22:53,083 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 12:23:00,478 [MainThread] INFO EvalMetric - ARand: 0.7455770573773435
2023-02-03 12:23:00,518 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 12:23:08,009 [MainThread] INFO EvalMetric - ARand: 0.7288633919006988
2023-02-03 12:23:08,049 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 12:23:15,484 [MainThread] INFO EvalMetric - ARand: 0.7131580566139526
2023-02-03 12:23:15,523 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 12:23:22,989 [MainThread] INFO EvalMetric - ARand: 0.6603636355963645
2023-02-03 12:23:23,032 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 12:23:30,437 [MainThread] INFO EvalMetric - ARand: 0.7250817067430839
2023-02-03 12:23:30,478 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 12:23:37,934 [MainThread] INFO EvalMetric - ARand: 0.7548375998068159
2023-02-03 12:23:37,973 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 12:23:45,396 [MainThread] INFO EvalMetric - ARand: 0.7059965916867237
2023-02-03 12:23:45,443 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 12:23:52,916 [MainThread] INFO EvalMetric - ARand: 0.7378142818255992
2023-02-03 12:23:52,953 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 12:24:00,600 [MainThread] INFO EvalMetric - ARand: 0.7203270729200164
2023-02-03 12:24:00,639 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 12:24:08,101 [MainThread] INFO EvalMetric - ARand: 0.7207594797427563
2023-02-03 12:24:08,142 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 12:24:15,570 [MainThread] INFO EvalMetric - ARand: 0.765239236198272
2023-02-03 12:24:15,618 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 12:24:23,161 [MainThread] INFO EvalMetric - ARand: 0.7422571358180075
2023-02-03 12:24:23,205 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 12:24:30,718 [MainThread] INFO EvalMetric - ARand: 0.7944765187506095
2023-02-03 12:24:30,763 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 12:24:36,421 [MainThread] INFO EvalMetric - ARand: 0.7670083038392669
2023-02-03 12:24:36,809 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4867518139981675. Evaluation score: 0.7355866748435066
2023-02-03 12:24:36,836 [MainThread] INFO UNet3DTrainer - Saving new best evaluation metric: 0.7355866748435066
2023-02-03 12:24:36,838 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 12:24:44,366 [MainThread] INFO EvalMetric - ARand: 0.7545820315894018
2023-02-03 12:24:44,386 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.2507976606488228. Evaluation score: 0.7545820315894018
2023-02-03 12:24:44,387 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 12:24:45,403 [MainThread] INFO UNet3DTrainer - Training iteration [301/60000]. Epoch [5/499]
2023-02-03 12:24:45,787 [MainThread] INFO UNet3DTrainer - Training iteration [302/60000]. Epoch [5/499]
2023-02-03 12:24:46,309 [MainThread] INFO UNet3DTrainer - Training iteration [303/60000]. Epoch [5/499]
2023-02-03 12:24:46,858 [MainThread] INFO UNet3DTrainer - Training iteration [304/60000]. Epoch [5/499]
2023-02-03 12:24:47,410 [MainThread] INFO UNet3DTrainer - Training iteration [305/60000]. Epoch [5/499]
2023-02-03 12:24:47,959 [MainThread] INFO UNet3DTrainer - Training iteration [306/60000]. Epoch [5/499]
2023-02-03 12:24:48,660 [MainThread] INFO UNet3DTrainer - Training iteration [307/60000]. Epoch [5/499]
2023-02-03 12:24:49,270 [MainThread] INFO UNet3DTrainer - Training iteration [308/60000]. Epoch [5/499]
2023-02-03 12:24:49,862 [MainThread] INFO UNet3DTrainer - Training iteration [309/60000]. Epoch [5/499]
2023-02-03 12:24:50,439 [MainThread] INFO UNet3DTrainer - Training iteration [310/60000]. Epoch [5/499]
2023-02-03 12:24:51,062 [MainThread] INFO UNet3DTrainer - Training iteration [311/60000]. Epoch [5/499]
2023-02-03 12:24:51,588 [MainThread] INFO UNet3DTrainer - Training iteration [312/60000]. Epoch [5/499]
2023-02-03 12:25:27,805 [MainThread] INFO UNet3DTrainer - Training iteration [313/60000]. Epoch [5/499]
2023-02-03 12:25:28,172 [MainThread] INFO UNet3DTrainer - Training iteration [314/60000]. Epoch [5/499]
2023-02-03 12:25:28,710 [MainThread] INFO UNet3DTrainer - Training iteration [315/60000]. Epoch [5/499]
2023-02-03 12:25:29,280 [MainThread] INFO UNet3DTrainer - Training iteration [316/60000]. Epoch [5/499]
2023-02-03 12:25:29,826 [MainThread] INFO UNet3DTrainer - Training iteration [317/60000]. Epoch [5/499]
2023-02-03 12:25:30,387 [MainThread] INFO UNet3DTrainer - Training iteration [318/60000]. Epoch [5/499]
2023-02-03 12:25:32,070 [MainThread] INFO UNet3DTrainer - Training iteration [319/60000]. Epoch [5/499]
2023-02-03 12:25:32,540 [MainThread] INFO UNet3DTrainer - Training iteration [320/60000]. Epoch [5/499]
2023-02-03 12:25:33,140 [MainThread] INFO UNet3DTrainer - Training iteration [321/60000]. Epoch [5/499]
2023-02-03 12:25:33,696 [MainThread] INFO UNet3DTrainer - Training iteration [322/60000]. Epoch [5/499]
2023-02-03 12:25:35,224 [MainThread] INFO UNet3DTrainer - Training iteration [323/60000]. Epoch [5/499]
2023-02-03 12:25:35,660 [MainThread] INFO UNet3DTrainer - Training iteration [324/60000]. Epoch [5/499]
2023-02-03 12:25:37,667 [MainThread] INFO UNet3DTrainer - Training iteration [325/60000]. Epoch [5/499]
2023-02-03 12:25:38,037 [MainThread] INFO UNet3DTrainer - Training iteration [326/60000]. Epoch [5/499]
2023-02-03 12:25:38,616 [MainThread] INFO UNet3DTrainer - Training iteration [327/60000]. Epoch [5/499]
2023-02-03 12:26:17,633 [MainThread] INFO UNet3DTrainer - Training iteration [328/60000]. Epoch [5/499]
2023-02-03 12:26:18,014 [MainThread] INFO UNet3DTrainer - Training iteration [329/60000]. Epoch [5/499]
2023-02-03 12:26:18,545 [MainThread] INFO UNet3DTrainer - Training iteration [330/60000]. Epoch [5/499]
2023-02-03 12:26:19,077 [MainThread] INFO UNet3DTrainer - Training iteration [331/60000]. Epoch [5/499]
2023-02-03 12:26:19,603 [MainThread] INFO UNet3DTrainer - Training iteration [332/60000]. Epoch [5/499]
2023-02-03 12:26:20,137 [MainThread] INFO UNet3DTrainer - Training iteration [333/60000]. Epoch [5/499]
2023-02-03 12:26:20,668 [MainThread] INFO UNet3DTrainer - Training iteration [334/60000]. Epoch [5/499]
2023-02-03 12:26:21,201 [MainThread] INFO UNet3DTrainer - Training iteration [335/60000]. Epoch [5/499]
2023-02-03 12:26:21,724 [MainThread] INFO UNet3DTrainer - Training iteration [336/60000]. Epoch [5/499]
2023-02-03 12:26:51,525 [MainThread] INFO UNet3DTrainer - Training iteration [337/60000]. Epoch [6/499]
2023-02-03 12:26:51,919 [MainThread] INFO UNet3DTrainer - Training iteration [338/60000]. Epoch [6/499]
2023-02-03 12:27:12,736 [MainThread] INFO UNet3DTrainer - Training iteration [339/60000]. Epoch [6/499]
2023-02-03 12:27:13,118 [MainThread] INFO UNet3DTrainer - Training iteration [340/60000]. Epoch [6/499]
2023-02-03 12:27:13,673 [MainThread] INFO UNet3DTrainer - Training iteration [341/60000]. Epoch [6/499]
2023-02-03 12:27:14,236 [MainThread] INFO UNet3DTrainer - Training iteration [342/60000]. Epoch [6/499]
2023-02-03 12:27:17,939 [MainThread] INFO UNet3DTrainer - Training iteration [343/60000]. Epoch [6/499]
2023-02-03 12:27:18,369 [MainThread] INFO UNet3DTrainer - Training iteration [344/60000]. Epoch [6/499]
2023-02-03 12:27:18,922 [MainThread] INFO UNet3DTrainer - Training iteration [345/60000]. Epoch [6/499]
2023-02-03 12:27:19,507 [MainThread] INFO UNet3DTrainer - Training iteration [346/60000]. Epoch [6/499]
2023-02-03 12:27:20,039 [MainThread] INFO UNet3DTrainer - Training iteration [347/60000]. Epoch [6/499]
2023-02-03 12:27:20,612 [MainThread] INFO UNet3DTrainer - Training iteration [348/60000]. Epoch [6/499]
2023-02-03 12:27:22,639 [MainThread] INFO UNet3DTrainer - Training iteration [349/60000]. Epoch [6/499]
2023-02-03 12:27:23,151 [MainThread] INFO UNet3DTrainer - Training iteration [350/60000]. Epoch [6/499]
2023-02-03 12:27:23,714 [MainThread] INFO UNet3DTrainer - Training iteration [351/60000]. Epoch [6/499]
2023-02-03 12:27:24,350 [MainThread] INFO UNet3DTrainer - Training iteration [352/60000]. Epoch [6/499]
2023-02-03 12:27:46,843 [MainThread] INFO UNet3DTrainer - Training iteration [353/60000]. Epoch [6/499]
2023-02-03 12:27:47,239 [MainThread] INFO UNet3DTrainer - Training iteration [354/60000]. Epoch [6/499]
2023-02-03 12:27:47,807 [MainThread] INFO UNet3DTrainer - Training iteration [355/60000]. Epoch [6/499]
2023-02-03 12:27:48,480 [MainThread] INFO UNet3DTrainer - Training iteration [356/60000]. Epoch [6/499]
2023-02-03 12:28:10,429 [MainThread] INFO UNet3DTrainer - Training iteration [357/60000]. Epoch [6/499]
2023-02-03 12:28:10,812 [MainThread] INFO UNet3DTrainer - Training iteration [358/60000]. Epoch [6/499]
2023-02-03 12:28:11,368 [MainThread] INFO UNet3DTrainer - Training iteration [359/60000]. Epoch [6/499]
2023-02-03 12:28:11,902 [MainThread] INFO UNet3DTrainer - Training iteration [360/60000]. Epoch [6/499]
2023-02-03 12:28:12,461 [MainThread] INFO UNet3DTrainer - Training iteration [361/60000]. Epoch [6/499]
2023-02-03 12:28:13,005 [MainThread] INFO UNet3DTrainer - Training iteration [362/60000]. Epoch [6/499]
2023-02-03 12:28:14,750 [MainThread] INFO UNet3DTrainer - Training iteration [363/60000]. Epoch [6/499]
2023-02-03 12:28:15,180 [MainThread] INFO UNet3DTrainer - Training iteration [364/60000]. Epoch [6/499]
2023-02-03 12:28:38,608 [MainThread] INFO UNet3DTrainer - Training iteration [365/60000]. Epoch [6/499]
2023-02-03 12:28:38,994 [MainThread] INFO UNet3DTrainer - Training iteration [366/60000]. Epoch [6/499]
2023-02-03 12:28:39,565 [MainThread] INFO UNet3DTrainer - Training iteration [367/60000]. Epoch [6/499]
2023-02-03 12:28:40,123 [MainThread] INFO UNet3DTrainer - Training iteration [368/60000]. Epoch [6/499]
2023-02-03 12:29:02,723 [MainThread] INFO UNet3DTrainer - Training iteration [369/60000]. Epoch [6/499]
2023-02-03 12:29:03,191 [MainThread] INFO UNet3DTrainer - Training iteration [370/60000]. Epoch [6/499]
2023-02-03 12:29:03,731 [MainThread] INFO UNet3DTrainer - Training iteration [371/60000]. Epoch [6/499]
2023-02-03 12:29:04,283 [MainThread] INFO UNet3DTrainer - Training iteration [372/60000]. Epoch [6/499]
2023-02-03 12:29:04,827 [MainThread] INFO UNet3DTrainer - Training iteration [373/60000]. Epoch [6/499]
2023-02-03 12:29:05,395 [MainThread] INFO UNet3DTrainer - Training iteration [374/60000]. Epoch [6/499]
2023-02-03 12:29:07,290 [MainThread] INFO UNet3DTrainer - Training iteration [375/60000]. Epoch [6/499]
2023-02-03 12:29:07,739 [MainThread] INFO UNet3DTrainer - Training iteration [376/60000]. Epoch [6/499]
2023-02-03 12:29:08,320 [MainThread] INFO UNet3DTrainer - Training iteration [377/60000]. Epoch [6/499]
2023-02-03 12:29:49,325 [MainThread] INFO UNet3DTrainer - Training iteration [378/60000]. Epoch [6/499]
2023-02-03 12:29:49,698 [MainThread] INFO UNet3DTrainer - Training iteration [379/60000]. Epoch [6/499]
2023-02-03 12:29:50,227 [MainThread] INFO UNet3DTrainer - Training iteration [380/60000]. Epoch [6/499]
2023-02-03 12:29:50,764 [MainThread] INFO UNet3DTrainer - Training iteration [381/60000]. Epoch [6/499]
2023-02-03 12:29:51,307 [MainThread] INFO UNet3DTrainer - Training iteration [382/60000]. Epoch [6/499]
2023-02-03 12:29:51,845 [MainThread] INFO UNet3DTrainer - Training iteration [383/60000]. Epoch [6/499]
2023-02-03 12:30:10,637 [MainThread] INFO UNet3DTrainer - Training iteration [384/60000]. Epoch [6/499]
2023-02-03 12:30:11,013 [MainThread] INFO UNet3DTrainer - Training iteration [385/60000]. Epoch [6/499]
2023-02-03 12:30:11,554 [MainThread] INFO UNet3DTrainer - Training iteration [386/60000]. Epoch [6/499]
2023-02-03 12:30:12,086 [MainThread] INFO UNet3DTrainer - Training iteration [387/60000]. Epoch [6/499]
2023-02-03 12:30:12,612 [MainThread] INFO UNet3DTrainer - Training iteration [388/60000]. Epoch [6/499]
2023-02-03 12:30:13,141 [MainThread] INFO UNet3DTrainer - Training iteration [389/60000]. Epoch [6/499]
2023-02-03 12:30:13,877 [MainThread] INFO UNet3DTrainer - Training iteration [390/60000]. Epoch [6/499]
2023-02-03 12:30:14,209 [MainThread] INFO UNet3DTrainer - Training iteration [391/60000]. Epoch [6/499]
2023-02-03 12:30:14,735 [MainThread] INFO UNet3DTrainer - Training iteration [392/60000]. Epoch [6/499]
2023-02-03 12:30:22,520 [MainThread] INFO UNet3DTrainer - Training iteration [393/60000]. Epoch [7/499]
2023-02-03 12:31:09,782 [MainThread] INFO UNet3DTrainer - Training iteration [394/60000]. Epoch [7/499]
2023-02-03 12:31:10,382 [MainThread] INFO UNet3DTrainer - Training iteration [395/60000]. Epoch [7/499]
2023-02-03 12:31:10,750 [MainThread] INFO UNet3DTrainer - Training iteration [396/60000]. Epoch [7/499]
2023-02-03 12:31:11,298 [MainThread] INFO UNet3DTrainer - Training iteration [397/60000]. Epoch [7/499]
2023-02-03 12:31:11,932 [MainThread] INFO UNet3DTrainer - Training iteration [398/60000]. Epoch [7/499]
2023-02-03 12:31:12,524 [MainThread] INFO UNet3DTrainer - Training iteration [399/60000]. Epoch [7/499]
2023-02-03 12:31:14,340 [MainThread] INFO UNet3DTrainer - Training iteration [400/60000]. Epoch [7/499]
2023-02-03 12:31:14,762 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 12:31:20,649 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 12:31:31,560 [MainThread] INFO EvalMetric - ARand: 0.6728893898111122
2023-02-03 12:31:31,607 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 12:31:39,992 [MainThread] INFO EvalMetric - ARand: 0.7899844175163553
2023-02-03 12:31:40,047 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 12:31:47,892 [MainThread] INFO EvalMetric - ARand: 0.8066554702552604
2023-02-03 12:31:47,935 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 12:31:55,581 [MainThread] INFO EvalMetric - ARand: 0.7283821176202939
2023-02-03 12:31:55,628 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 12:32:03,246 [MainThread] INFO EvalMetric - ARand: 0.7234474403638073
2023-02-03 12:32:03,289 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 12:32:10,758 [MainThread] INFO EvalMetric - ARand: 0.7708925479000059
2023-02-03 12:32:10,811 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 12:32:18,172 [MainThread] INFO EvalMetric - ARand: 0.6787275105323297
2023-02-03 12:32:18,212 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 12:32:25,606 [MainThread] INFO EvalMetric - ARand: 0.7535704089545504
2023-02-03 12:32:25,653 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 12:32:33,132 [MainThread] INFO EvalMetric - ARand: 0.7241249373228424
2023-02-03 12:32:33,178 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 12:32:40,534 [MainThread] INFO EvalMetric - ARand: 0.7538019330720563
2023-02-03 12:32:40,571 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 12:32:48,073 [MainThread] INFO EvalMetric - ARand: 0.681867017064518
2023-02-03 12:32:48,113 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 12:32:55,571 [MainThread] INFO EvalMetric - ARand: 0.662079977459948
2023-02-03 12:32:55,612 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 12:33:02,996 [MainThread] INFO EvalMetric - ARand: 0.7324660358202566
2023-02-03 12:33:03,033 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 12:33:10,427 [MainThread] INFO EvalMetric - ARand: 0.7843681304649
2023-02-03 12:33:10,465 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 12:33:17,854 [MainThread] INFO EvalMetric - ARand: 0.688596108057853
2023-02-03 12:33:17,896 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 12:33:25,329 [MainThread] INFO EvalMetric - ARand: 0.7303095751113851
2023-02-03 12:33:25,374 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 12:33:32,892 [MainThread] INFO EvalMetric - ARand: 0.7547430743261282
2023-02-03 12:33:32,933 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 12:33:40,321 [MainThread] INFO EvalMetric - ARand: 0.7298503354224879
2023-02-03 12:33:40,369 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 12:33:47,829 [MainThread] INFO EvalMetric - ARand: 0.7874772228452336
2023-02-03 12:33:47,874 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 12:33:55,267 [MainThread] INFO EvalMetric - ARand: 0.7652709677142746
2023-02-03 12:33:55,315 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 12:34:02,859 [MainThread] INFO EvalMetric - ARand: 0.8195738704526425
2023-02-03 12:34:02,906 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 12:34:08,551 [MainThread] INFO EvalMetric - ARand: 0.8128966433744033
2023-02-03 12:34:08,978 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4847217647508644. Evaluation score: 0.7424713089939787
2023-02-03 12:34:09,006 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 12:34:16,410 [MainThread] INFO EvalMetric - ARand: 0.7080725122747318
2023-02-03 12:34:16,429 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.23604718782007694. Evaluation score: 0.7080725122747318
2023-02-03 12:34:16,431 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 12:34:17,492 [MainThread] INFO UNet3DTrainer - Training iteration [401/60000]. Epoch [7/499]
2023-02-03 12:34:17,889 [MainThread] INFO UNet3DTrainer - Training iteration [402/60000]. Epoch [7/499]
2023-02-03 12:34:18,419 [MainThread] INFO UNet3DTrainer - Training iteration [403/60000]. Epoch [7/499]
2023-02-03 12:34:18,961 [MainThread] INFO UNet3DTrainer - Training iteration [404/60000]. Epoch [7/499]
2023-02-03 12:34:19,505 [MainThread] INFO UNet3DTrainer - Training iteration [405/60000]. Epoch [7/499]
2023-02-03 12:34:20,070 [MainThread] INFO UNet3DTrainer - Training iteration [406/60000]. Epoch [7/499]
2023-02-03 12:34:20,677 [MainThread] INFO UNet3DTrainer - Training iteration [407/60000]. Epoch [7/499]
2023-02-03 12:34:21,277 [MainThread] INFO UNet3DTrainer - Training iteration [408/60000]. Epoch [7/499]
2023-02-03 12:34:21,880 [MainThread] INFO UNet3DTrainer - Training iteration [409/60000]. Epoch [7/499]
2023-02-03 12:34:22,430 [MainThread] INFO UNet3DTrainer - Training iteration [410/60000]. Epoch [7/499]
2023-02-03 12:34:23,150 [MainThread] INFO UNet3DTrainer - Training iteration [411/60000]. Epoch [7/499]
2023-02-03 12:34:23,700 [MainThread] INFO UNet3DTrainer - Training iteration [412/60000]. Epoch [7/499]
2023-02-03 12:34:24,247 [MainThread] INFO UNet3DTrainer - Training iteration [413/60000]. Epoch [7/499]
2023-02-03 12:34:46,839 [MainThread] INFO UNet3DTrainer - Training iteration [414/60000]. Epoch [7/499]
2023-02-03 12:34:47,224 [MainThread] INFO UNet3DTrainer - Training iteration [415/60000]. Epoch [7/499]
2023-02-03 12:34:47,783 [MainThread] INFO UNet3DTrainer - Training iteration [416/60000]. Epoch [7/499]
2023-02-03 12:34:48,344 [MainThread] INFO UNet3DTrainer - Training iteration [417/60000]. Epoch [7/499]
2023-02-03 12:34:48,948 [MainThread] INFO UNet3DTrainer - Training iteration [418/60000]. Epoch [7/499]
2023-02-03 12:34:50,669 [MainThread] INFO UNet3DTrainer - Training iteration [419/60000]. Epoch [7/499]
2023-02-03 12:34:51,585 [MainThread] INFO UNet3DTrainer - Training iteration [420/60000]. Epoch [7/499]
2023-02-03 12:34:52,090 [MainThread] INFO UNet3DTrainer - Training iteration [421/60000]. Epoch [7/499]
2023-02-03 12:34:52,942 [MainThread] INFO UNet3DTrainer - Training iteration [422/60000]. Epoch [7/499]
2023-02-03 12:34:53,357 [MainThread] INFO UNet3DTrainer - Training iteration [423/60000]. Epoch [7/499]
2023-02-03 12:35:21,157 [MainThread] INFO UNet3DTrainer - Training iteration [424/60000]. Epoch [7/499]
2023-02-03 12:35:21,693 [MainThread] INFO UNet3DTrainer - Training iteration [425/60000]. Epoch [7/499]
2023-02-03 12:35:22,268 [MainThread] INFO UNet3DTrainer - Training iteration [426/60000]. Epoch [7/499]
2023-02-03 12:35:22,910 [MainThread] INFO UNet3DTrainer - Training iteration [427/60000]. Epoch [7/499]
2023-02-03 12:35:23,480 [MainThread] INFO UNet3DTrainer - Training iteration [428/60000]. Epoch [7/499]
2023-02-03 12:35:24,061 [MainThread] INFO UNet3DTrainer - Training iteration [429/60000]. Epoch [7/499]
2023-02-03 12:35:26,260 [MainThread] INFO UNet3DTrainer - Training iteration [430/60000]. Epoch [7/499]
2023-02-03 12:35:26,717 [MainThread] INFO UNet3DTrainer - Training iteration [431/60000]. Epoch [7/499]
2023-02-03 12:35:27,991 [MainThread] INFO UNet3DTrainer - Training iteration [432/60000]. Epoch [7/499]
2023-02-03 12:35:50,650 [MainThread] INFO UNet3DTrainer - Training iteration [433/60000]. Epoch [7/499]
2023-02-03 12:35:51,050 [MainThread] INFO UNet3DTrainer - Training iteration [434/60000]. Epoch [7/499]
2023-02-03 12:35:51,602 [MainThread] INFO UNet3DTrainer - Training iteration [435/60000]. Epoch [7/499]
2023-02-03 12:35:52,156 [MainThread] INFO UNet3DTrainer - Training iteration [436/60000]. Epoch [7/499]
2023-02-03 12:35:52,714 [MainThread] INFO UNet3DTrainer - Training iteration [437/60000]. Epoch [7/499]
2023-02-03 12:35:54,832 [MainThread] INFO UNet3DTrainer - Training iteration [438/60000]. Epoch [7/499]
2023-02-03 12:35:55,277 [MainThread] INFO UNet3DTrainer - Training iteration [439/60000]. Epoch [7/499]
2023-02-03 12:36:12,487 [MainThread] INFO UNet3DTrainer - Training iteration [440/60000]. Epoch [7/499]
2023-02-03 12:36:12,870 [MainThread] INFO UNet3DTrainer - Training iteration [441/60000]. Epoch [7/499]
2023-02-03 12:36:13,419 [MainThread] INFO UNet3DTrainer - Training iteration [442/60000]. Epoch [7/499]
2023-02-03 12:36:13,956 [MainThread] INFO UNet3DTrainer - Training iteration [443/60000]. Epoch [7/499]
2023-02-03 12:36:34,886 [MainThread] INFO UNet3DTrainer - Training iteration [444/60000]. Epoch [7/499]
2023-02-03 12:36:35,264 [MainThread] INFO UNet3DTrainer - Training iteration [445/60000]. Epoch [7/499]
2023-02-03 12:36:35,793 [MainThread] INFO UNet3DTrainer - Training iteration [446/60000]. Epoch [7/499]
2023-02-03 12:36:36,326 [MainThread] INFO UNet3DTrainer - Training iteration [447/60000]. Epoch [7/499]
2023-02-03 12:36:36,853 [MainThread] INFO UNet3DTrainer - Training iteration [448/60000]. Epoch [7/499]
2023-02-03 12:37:05,689 [MainThread] INFO UNet3DTrainer - Training iteration [449/60000]. Epoch [8/499]
2023-02-03 12:37:06,076 [MainThread] INFO UNet3DTrainer - Training iteration [450/60000]. Epoch [8/499]
2023-02-03 12:37:06,627 [MainThread] INFO UNet3DTrainer - Training iteration [451/60000]. Epoch [8/499]
2023-02-03 12:37:07,183 [MainThread] INFO UNet3DTrainer - Training iteration [452/60000]. Epoch [8/499]
2023-02-03 12:37:07,760 [MainThread] INFO UNet3DTrainer - Training iteration [453/60000]. Epoch [8/499]
2023-02-03 12:37:08,403 [MainThread] INFO UNet3DTrainer - Training iteration [454/60000]. Epoch [8/499]
2023-02-03 12:37:52,264 [MainThread] INFO UNet3DTrainer - Training iteration [455/60000]. Epoch [8/499]
2023-02-03 12:37:52,635 [MainThread] INFO UNet3DTrainer - Training iteration [456/60000]. Epoch [8/499]
2023-02-03 12:37:53,164 [MainThread] INFO UNet3DTrainer - Training iteration [457/60000]. Epoch [8/499]
2023-02-03 12:37:53,712 [MainThread] INFO UNet3DTrainer - Training iteration [458/60000]. Epoch [8/499]
2023-02-03 12:37:54,272 [MainThread] INFO UNet3DTrainer - Training iteration [459/60000]. Epoch [8/499]
2023-02-03 12:37:54,930 [MainThread] INFO UNet3DTrainer - Training iteration [460/60000]. Epoch [8/499]
2023-02-03 12:37:56,410 [MainThread] INFO UNet3DTrainer - Training iteration [461/60000]. Epoch [8/499]
2023-02-03 12:37:56,911 [MainThread] INFO UNet3DTrainer - Training iteration [462/60000]. Epoch [8/499]
2023-02-03 12:37:57,465 [MainThread] INFO UNet3DTrainer - Training iteration [463/60000]. Epoch [8/499]
2023-02-03 12:37:58,401 [MainThread] INFO UNet3DTrainer - Training iteration [464/60000]. Epoch [8/499]
2023-02-03 12:37:58,879 [MainThread] INFO UNet3DTrainer - Training iteration [465/60000]. Epoch [8/499]
2023-02-03 12:37:59,532 [MainThread] INFO UNet3DTrainer - Training iteration [466/60000]. Epoch [8/499]
2023-02-03 12:38:02,116 [MainThread] INFO UNet3DTrainer - Training iteration [467/60000]. Epoch [8/499]
2023-02-03 12:38:02,600 [MainThread] INFO UNet3DTrainer - Training iteration [468/60000]. Epoch [8/499]
2023-02-03 12:38:03,320 [MainThread] INFO UNet3DTrainer - Training iteration [469/60000]. Epoch [8/499]
2023-02-03 12:38:03,910 [MainThread] INFO UNet3DTrainer - Training iteration [470/60000]. Epoch [8/499]
2023-02-03 12:38:21,585 [MainThread] INFO UNet3DTrainer - Training iteration [471/60000]. Epoch [8/499]
2023-02-03 12:38:21,963 [MainThread] INFO UNet3DTrainer - Training iteration [472/60000]. Epoch [8/499]
2023-02-03 12:38:22,522 [MainThread] INFO UNet3DTrainer - Training iteration [473/60000]. Epoch [8/499]
2023-02-03 12:38:23,080 [MainThread] INFO UNet3DTrainer - Training iteration [474/60000]. Epoch [8/499]
2023-02-03 12:38:23,634 [MainThread] INFO UNet3DTrainer - Training iteration [475/60000]. Epoch [8/499]
2023-02-03 12:38:30,057 [MainThread] INFO UNet3DTrainer - Training iteration [476/60000]. Epoch [8/499]
2023-02-03 12:38:47,169 [MainThread] INFO UNet3DTrainer - Training iteration [477/60000]. Epoch [8/499]
2023-02-03 12:38:47,553 [MainThread] INFO UNet3DTrainer - Training iteration [478/60000]. Epoch [8/499]
2023-02-03 12:38:55,071 [MainThread] INFO UNet3DTrainer - Training iteration [479/60000]. Epoch [8/499]
2023-02-03 12:38:55,455 [MainThread] INFO UNet3DTrainer - Training iteration [480/60000]. Epoch [8/499]
2023-02-03 12:38:56,023 [MainThread] INFO UNet3DTrainer - Training iteration [481/60000]. Epoch [8/499]
2023-02-03 12:38:56,618 [MainThread] INFO UNet3DTrainer - Training iteration [482/60000]. Epoch [8/499]
2023-02-03 12:38:57,189 [MainThread] INFO UNet3DTrainer - Training iteration [483/60000]. Epoch [8/499]
2023-02-03 12:38:57,754 [MainThread] INFO UNet3DTrainer - Training iteration [484/60000]. Epoch [8/499]
2023-02-03 12:39:00,151 [MainThread] INFO UNet3DTrainer - Training iteration [485/60000]. Epoch [8/499]
2023-02-03 12:39:00,660 [MainThread] INFO UNet3DTrainer - Training iteration [486/60000]. Epoch [8/499]
2023-02-03 12:39:01,202 [MainThread] INFO UNet3DTrainer - Training iteration [487/60000]. Epoch [8/499]
2023-02-03 12:39:01,880 [MainThread] INFO UNet3DTrainer - Training iteration [488/60000]. Epoch [8/499]
2023-02-03 12:39:21,145 [MainThread] INFO UNet3DTrainer - Training iteration [489/60000]. Epoch [8/499]
2023-02-03 12:39:21,534 [MainThread] INFO UNet3DTrainer - Training iteration [490/60000]. Epoch [8/499]
2023-02-03 12:39:53,180 [MainThread] INFO UNet3DTrainer - Training iteration [491/60000]. Epoch [8/499]
2023-02-03 12:39:53,558 [MainThread] INFO UNet3DTrainer - Training iteration [492/60000]. Epoch [8/499]
2023-02-03 12:39:54,112 [MainThread] INFO UNet3DTrainer - Training iteration [493/60000]. Epoch [8/499]
2023-02-03 12:39:54,670 [MainThread] INFO UNet3DTrainer - Training iteration [494/60000]. Epoch [8/499]
2023-02-03 12:39:55,202 [MainThread] INFO UNet3DTrainer - Training iteration [495/60000]. Epoch [8/499]
2023-02-03 12:39:55,767 [MainThread] INFO UNet3DTrainer - Training iteration [496/60000]. Epoch [8/499]
2023-02-03 12:39:57,321 [MainThread] INFO UNet3DTrainer - Training iteration [497/60000]. Epoch [8/499]
2023-02-03 12:39:57,684 [MainThread] INFO UNet3DTrainer - Training iteration [498/60000]. Epoch [8/499]
2023-02-03 12:39:58,209 [MainThread] INFO UNet3DTrainer - Training iteration [499/60000]. Epoch [8/499]
2023-02-03 12:39:58,746 [MainThread] INFO UNet3DTrainer - Training iteration [500/60000]. Epoch [8/499]
2023-02-03 12:39:59,259 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 12:40:04,080 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 12:40:13,416 [MainThread] INFO EvalMetric - ARand: 0.666506316920837
2023-02-03 12:40:13,462 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 12:40:21,025 [MainThread] INFO EvalMetric - ARand: 0.7800788394593015
2023-02-03 12:40:21,074 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 12:40:28,798 [MainThread] INFO EvalMetric - ARand: 0.7981211899442188
2023-02-03 12:40:28,838 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 12:40:36,267 [MainThread] INFO EvalMetric - ARand: 0.7465669210931497
2023-02-03 12:40:36,303 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 12:40:43,830 [MainThread] INFO EvalMetric - ARand: 0.7232900518618164
2023-02-03 12:40:43,870 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 12:40:51,255 [MainThread] INFO EvalMetric - ARand: 0.794592480800919
2023-02-03 12:40:51,297 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 12:40:58,714 [MainThread] INFO EvalMetric - ARand: 0.6898662850052905
2023-02-03 12:40:58,761 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 12:41:06,126 [MainThread] INFO EvalMetric - ARand: 0.7557155777499462
2023-02-03 12:41:06,165 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 12:41:13,514 [MainThread] INFO EvalMetric - ARand: 0.7411833680616294
2023-02-03 12:41:13,555 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 12:41:20,998 [MainThread] INFO EvalMetric - ARand: 0.7464703251173608
2023-02-03 12:41:21,039 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 12:41:28,546 [MainThread] INFO EvalMetric - ARand: 0.7085967966397873
2023-02-03 12:41:28,585 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 12:41:36,003 [MainThread] INFO EvalMetric - ARand: 0.6626194665920815
2023-02-03 12:41:36,052 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 12:41:43,451 [MainThread] INFO EvalMetric - ARand: 0.7160500421965942
2023-02-03 12:41:43,488 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 12:41:50,880 [MainThread] INFO EvalMetric - ARand: 0.7784141584843813
2023-02-03 12:41:50,922 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 12:41:58,451 [MainThread] INFO EvalMetric - ARand: 0.7027297226902351
2023-02-03 12:41:58,493 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 12:42:05,914 [MainThread] INFO EvalMetric - ARand: 0.7186381347736468
2023-02-03 12:42:05,961 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 12:42:13,445 [MainThread] INFO EvalMetric - ARand: 0.7453940125448063
2023-02-03 12:42:13,490 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 12:42:21,026 [MainThread] INFO EvalMetric - ARand: 0.765764842698859
2023-02-03 12:42:21,069 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 12:42:28,478 [MainThread] INFO EvalMetric - ARand: 0.7495722027419043
2023-02-03 12:42:28,522 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 12:42:35,974 [MainThread] INFO EvalMetric - ARand: 0.739393422667705
2023-02-03 12:42:36,017 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 12:42:43,416 [MainThread] INFO EvalMetric - ARand: 0.7860822071011788
2023-02-03 12:42:43,461 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 12:42:49,077 [MainThread] INFO EvalMetric - ARand: 0.7726945078737311
2023-02-03 12:42:49,464 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4824098293808685. Evaluation score: 0.7400076894736068
2023-02-03 12:42:49,490 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 12:42:56,679 [MainThread] INFO EvalMetric - ARand: 0.7736770350520363
2023-02-03 12:42:56,698 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.22559971305040213. Evaluation score: 0.7736770350520363
2023-02-03 12:42:56,699 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 12:42:57,749 [MainThread] INFO UNet3DTrainer - Training iteration [501/60000]. Epoch [8/499]
2023-02-03 12:42:58,123 [MainThread] INFO UNet3DTrainer - Training iteration [502/60000]. Epoch [8/499]
2023-02-03 12:42:58,651 [MainThread] INFO UNet3DTrainer - Training iteration [503/60000]. Epoch [8/499]
2023-02-03 12:42:59,179 [MainThread] INFO UNet3DTrainer - Training iteration [504/60000]. Epoch [8/499]
2023-02-03 12:43:06,981 [MainThread] INFO UNet3DTrainer - Training iteration [505/60000]. Epoch [9/499]
2023-02-03 12:43:28,161 [MainThread] INFO UNet3DTrainer - Training iteration [506/60000]. Epoch [9/499]
2023-02-03 12:43:28,561 [MainThread] INFO UNet3DTrainer - Training iteration [507/60000]. Epoch [9/499]
2023-02-03 12:43:29,124 [MainThread] INFO UNet3DTrainer - Training iteration [508/60000]. Epoch [9/499]
2023-02-03 12:43:29,727 [MainThread] INFO UNet3DTrainer - Training iteration [509/60000]. Epoch [9/499]
2023-02-03 12:43:30,359 [MainThread] INFO UNet3DTrainer - Training iteration [510/60000]. Epoch [9/499]
2023-02-03 12:43:30,965 [MainThread] INFO UNet3DTrainer - Training iteration [511/60000]. Epoch [9/499]
2023-02-03 12:43:32,763 [MainThread] INFO UNet3DTrainer - Training iteration [512/60000]. Epoch [9/499]
2023-02-03 12:43:33,263 [MainThread] INFO UNet3DTrainer - Training iteration [513/60000]. Epoch [9/499]
2023-02-03 12:43:33,878 [MainThread] INFO UNet3DTrainer - Training iteration [514/60000]. Epoch [9/499]
2023-02-03 12:43:34,470 [MainThread] INFO UNet3DTrainer - Training iteration [515/60000]. Epoch [9/499]
2023-02-03 12:43:54,682 [MainThread] INFO UNet3DTrainer - Training iteration [516/60000]. Epoch [9/499]
2023-02-03 12:43:55,086 [MainThread] INFO UNet3DTrainer - Training iteration [517/60000]. Epoch [9/499]
2023-02-03 12:44:16,184 [MainThread] INFO UNet3DTrainer - Training iteration [518/60000]. Epoch [9/499]
2023-02-03 12:44:16,563 [MainThread] INFO UNet3DTrainer - Training iteration [519/60000]. Epoch [9/499]
2023-02-03 12:44:17,086 [MainThread] INFO UNet3DTrainer - Training iteration [520/60000]. Epoch [9/499]
2023-02-03 12:44:17,649 [MainThread] INFO UNet3DTrainer - Training iteration [521/60000]. Epoch [9/499]
2023-02-03 12:44:18,191 [MainThread] INFO UNet3DTrainer - Training iteration [522/60000]. Epoch [9/499]
2023-02-03 12:44:18,780 [MainThread] INFO UNet3DTrainer - Training iteration [523/60000]. Epoch [9/499]
2023-02-03 12:44:20,108 [MainThread] INFO UNet3DTrainer - Training iteration [524/60000]. Epoch [9/499]
2023-02-03 12:44:20,606 [MainThread] INFO UNet3DTrainer - Training iteration [525/60000]. Epoch [9/499]
2023-02-03 12:44:21,200 [MainThread] INFO UNet3DTrainer - Training iteration [526/60000]. Epoch [9/499]
2023-02-03 12:44:21,780 [MainThread] INFO UNet3DTrainer - Training iteration [527/60000]. Epoch [9/499]
2023-02-03 12:44:22,363 [MainThread] INFO UNet3DTrainer - Training iteration [528/60000]. Epoch [9/499]
2023-02-03 12:44:23,030 [MainThread] INFO UNet3DTrainer - Training iteration [529/60000]. Epoch [9/499]
2023-02-03 12:44:50,672 [MainThread] INFO UNet3DTrainer - Training iteration [530/60000]. Epoch [9/499]
2023-02-03 12:44:51,172 [MainThread] INFO UNet3DTrainer - Training iteration [531/60000]. Epoch [9/499]
2023-02-03 12:44:51,728 [MainThread] INFO UNet3DTrainer - Training iteration [532/60000]. Epoch [9/499]
2023-02-03 12:44:52,330 [MainThread] INFO UNet3DTrainer - Training iteration [533/60000]. Epoch [9/499]
2023-02-03 12:44:52,905 [MainThread] INFO UNet3DTrainer - Training iteration [534/60000]. Epoch [9/499]
2023-02-03 12:44:53,462 [MainThread] INFO UNet3DTrainer - Training iteration [535/60000]. Epoch [9/499]
2023-02-03 12:45:19,521 [MainThread] INFO UNet3DTrainer - Training iteration [536/60000]. Epoch [9/499]
2023-02-03 12:45:35,224 [MainThread] INFO UNet3DTrainer - Training iteration [537/60000]. Epoch [9/499]
2023-02-03 12:45:35,597 [MainThread] INFO UNet3DTrainer - Training iteration [538/60000]. Epoch [9/499]
2023-02-03 12:45:36,156 [MainThread] INFO UNet3DTrainer - Training iteration [539/60000]. Epoch [9/499]
2023-02-03 12:45:37,138 [MainThread] INFO UNet3DTrainer - Training iteration [540/60000]. Epoch [9/499]
2023-02-03 12:45:37,571 [MainThread] INFO UNet3DTrainer - Training iteration [541/60000]. Epoch [9/499]
2023-02-03 12:45:38,131 [MainThread] INFO UNet3DTrainer - Training iteration [542/60000]. Epoch [9/499]
2023-02-03 12:45:40,428 [MainThread] INFO UNet3DTrainer - Training iteration [543/60000]. Epoch [9/499]
2023-02-03 12:45:40,906 [MainThread] INFO UNet3DTrainer - Training iteration [544/60000]. Epoch [9/499]
2023-02-03 12:45:41,506 [MainThread] INFO UNet3DTrainer - Training iteration [545/60000]. Epoch [9/499]
2023-02-03 12:45:42,462 [MainThread] INFO UNet3DTrainer - Training iteration [546/60000]. Epoch [9/499]
2023-02-03 12:45:42,863 [MainThread] INFO UNet3DTrainer - Training iteration [547/60000]. Epoch [9/499]
2023-02-03 12:45:48,831 [MainThread] INFO UNet3DTrainer - Training iteration [548/60000]. Epoch [9/499]
2023-02-03 12:46:09,480 [MainThread] INFO UNet3DTrainer - Training iteration [549/60000]. Epoch [9/499]
2023-02-03 12:46:09,888 [MainThread] INFO UNet3DTrainer - Training iteration [550/60000]. Epoch [9/499]
2023-02-03 12:46:10,441 [MainThread] INFO UNet3DTrainer - Training iteration [551/60000]. Epoch [9/499]
2023-02-03 12:46:11,010 [MainThread] INFO UNet3DTrainer - Training iteration [552/60000]. Epoch [9/499]
2023-02-03 12:46:27,396 [MainThread] INFO UNet3DTrainer - Training iteration [553/60000]. Epoch [9/499]
2023-02-03 12:46:27,772 [MainThread] INFO UNet3DTrainer - Training iteration [554/60000]. Epoch [9/499]
2023-02-03 12:46:28,317 [MainThread] INFO UNet3DTrainer - Training iteration [555/60000]. Epoch [9/499]
2023-02-03 12:46:29,058 [MainThread] INFO UNet3DTrainer - Training iteration [556/60000]. Epoch [9/499]
2023-02-03 12:46:30,692 [MainThread] INFO UNet3DTrainer - Training iteration [557/60000]. Epoch [9/499]
2023-02-03 12:46:31,019 [MainThread] INFO UNet3DTrainer - Training iteration [558/60000]. Epoch [9/499]
2023-02-03 12:47:07,646 [MainThread] INFO UNet3DTrainer - Training iteration [559/60000]. Epoch [9/499]
2023-02-03 12:47:08,013 [MainThread] INFO UNet3DTrainer - Training iteration [560/60000]. Epoch [9/499]
2023-02-03 12:47:58,980 [MainThread] INFO UNet3DTrainer - Training iteration [561/60000]. Epoch [10/499]
2023-02-03 12:47:59,357 [MainThread] INFO UNet3DTrainer - Training iteration [562/60000]. Epoch [10/499]
2023-02-03 12:47:59,897 [MainThread] INFO UNet3DTrainer - Training iteration [563/60000]. Epoch [10/499]
2023-02-03 12:48:00,447 [MainThread] INFO UNet3DTrainer - Training iteration [564/60000]. Epoch [10/499]
2023-02-03 12:48:01,002 [MainThread] INFO UNet3DTrainer - Training iteration [565/60000]. Epoch [10/499]
2023-02-03 12:48:01,557 [MainThread] INFO UNet3DTrainer - Training iteration [566/60000]. Epoch [10/499]
2023-02-03 12:48:03,398 [MainThread] INFO UNet3DTrainer - Training iteration [567/60000]. Epoch [10/499]
2023-02-03 12:48:24,726 [MainThread] INFO UNet3DTrainer - Training iteration [568/60000]. Epoch [10/499]
2023-02-03 12:48:25,112 [MainThread] INFO UNet3DTrainer - Training iteration [569/60000]. Epoch [10/499]
2023-02-03 12:48:25,678 [MainThread] INFO UNet3DTrainer - Training iteration [570/60000]. Epoch [10/499]
2023-02-03 12:48:26,239 [MainThread] INFO UNet3DTrainer - Training iteration [571/60000]. Epoch [10/499]
2023-02-03 12:48:26,847 [MainThread] INFO UNet3DTrainer - Training iteration [572/60000]. Epoch [10/499]
2023-02-03 12:48:52,171 [MainThread] INFO UNet3DTrainer - Training iteration [573/60000]. Epoch [10/499]
2023-02-03 12:48:52,555 [MainThread] INFO UNet3DTrainer - Training iteration [574/60000]. Epoch [10/499]
2023-02-03 12:48:53,087 [MainThread] INFO UNet3DTrainer - Training iteration [575/60000]. Epoch [10/499]
2023-02-03 12:48:53,621 [MainThread] INFO UNet3DTrainer - Training iteration [576/60000]. Epoch [10/499]
2023-02-03 12:48:54,183 [MainThread] INFO UNet3DTrainer - Training iteration [577/60000]. Epoch [10/499]
2023-02-03 12:48:54,745 [MainThread] INFO UNet3DTrainer - Training iteration [578/60000]. Epoch [10/499]
2023-02-03 12:49:33,566 [MainThread] INFO UNet3DTrainer - Training iteration [579/60000]. Epoch [10/499]
2023-02-03 12:49:33,934 [MainThread] INFO UNet3DTrainer - Training iteration [580/60000]. Epoch [10/499]
2023-02-03 12:49:34,471 [MainThread] INFO UNet3DTrainer - Training iteration [581/60000]. Epoch [10/499]
2023-02-03 12:49:34,997 [MainThread] INFO UNet3DTrainer - Training iteration [582/60000]. Epoch [10/499]
2023-02-03 12:49:35,549 [MainThread] INFO UNet3DTrainer - Training iteration [583/60000]. Epoch [10/499]
2023-02-03 12:49:36,106 [MainThread] INFO UNet3DTrainer - Training iteration [584/60000]. Epoch [10/499]
2023-02-03 12:49:38,054 [MainThread] INFO UNet3DTrainer - Training iteration [585/60000]. Epoch [10/499]
2023-02-03 12:49:38,513 [MainThread] INFO UNet3DTrainer - Training iteration [586/60000]. Epoch [10/499]
2023-02-03 12:49:39,150 [MainThread] INFO UNet3DTrainer - Training iteration [587/60000]. Epoch [10/499]
2023-02-03 12:49:39,703 [MainThread] INFO UNet3DTrainer - Training iteration [588/60000]. Epoch [10/499]
2023-02-03 12:49:40,300 [MainThread] INFO UNet3DTrainer - Training iteration [589/60000]. Epoch [10/499]
2023-02-03 12:49:40,872 [MainThread] INFO UNet3DTrainer - Training iteration [590/60000]. Epoch [10/499]
2023-02-03 12:50:03,991 [MainThread] INFO UNet3DTrainer - Training iteration [591/60000]. Epoch [10/499]
2023-02-03 12:50:04,382 [MainThread] INFO UNet3DTrainer - Training iteration [592/60000]. Epoch [10/499]
2023-02-03 12:50:21,878 [MainThread] INFO UNet3DTrainer - Training iteration [593/60000]. Epoch [10/499]
2023-02-03 12:50:22,246 [MainThread] INFO UNet3DTrainer - Training iteration [594/60000]. Epoch [10/499]
2023-02-03 12:50:22,804 [MainThread] INFO UNet3DTrainer - Training iteration [595/60000]. Epoch [10/499]
2023-02-03 12:50:23,347 [MainThread] INFO UNet3DTrainer - Training iteration [596/60000]. Epoch [10/499]
2023-02-03 12:50:23,908 [MainThread] INFO UNet3DTrainer - Training iteration [597/60000]. Epoch [10/499]
2023-02-03 12:50:24,530 [MainThread] INFO UNet3DTrainer - Training iteration [598/60000]. Epoch [10/499]
2023-02-03 12:50:26,357 [MainThread] INFO UNet3DTrainer - Training iteration [599/60000]. Epoch [10/499]
2023-02-03 12:50:26,755 [MainThread] INFO UNet3DTrainer - Training iteration [600/60000]. Epoch [10/499]
2023-02-03 12:50:27,286 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 12:50:33,621 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 12:50:44,751 [MainThread] INFO EvalMetric - ARand: 0.6390738729470087
2023-02-03 12:50:44,800 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 12:50:53,941 [MainThread] INFO EvalMetric - ARand: 0.7168550269989334
2023-02-03 12:50:54,000 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 12:51:02,802 [MainThread] INFO EvalMetric - ARand: 0.735162991872478
2023-02-03 12:51:02,843 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 12:51:10,395 [MainThread] INFO EvalMetric - ARand: 0.6847755308621762
2023-02-03 12:51:10,438 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 12:51:17,988 [MainThread] INFO EvalMetric - ARand: 0.6946879753086224
2023-02-03 12:51:18,030 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 12:51:25,640 [MainThread] INFO EvalMetric - ARand: 0.7672558692290842
2023-02-03 12:51:25,683 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 12:51:33,173 [MainThread] INFO EvalMetric - ARand: 0.6156085105107
2023-02-03 12:51:33,213 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 12:51:40,475 [MainThread] INFO EvalMetric - ARand: 0.7354331397929684
2023-02-03 12:51:40,516 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 12:51:47,877 [MainThread] INFO EvalMetric - ARand: 0.6805115225041553
2023-02-03 12:51:47,913 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 12:51:55,549 [MainThread] INFO EvalMetric - ARand: 0.7060120368834714
2023-02-03 12:51:55,587 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 12:52:03,075 [MainThread] INFO EvalMetric - ARand: 0.6644550299415326
2023-02-03 12:52:03,114 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 12:52:10,526 [MainThread] INFO EvalMetric - ARand: 0.6308960051916257
2023-02-03 12:52:10,566 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 12:52:17,978 [MainThread] INFO EvalMetric - ARand: 0.7119941458754508
2023-02-03 12:52:18,016 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 12:52:25,444 [MainThread] INFO EvalMetric - ARand: 0.7296722914475526
2023-02-03 12:52:25,486 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 12:52:32,996 [MainThread] INFO EvalMetric - ARand: 0.6799384034396461
2023-02-03 12:52:33,043 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 12:52:40,578 [MainThread] INFO EvalMetric - ARand: 0.6694468163587373
2023-02-03 12:52:40,618 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 12:52:48,135 [MainThread] INFO EvalMetric - ARand: 0.6495056194344355
2023-02-03 12:52:48,181 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 12:52:55,713 [MainThread] INFO EvalMetric - ARand: 0.7308744935461524
2023-02-03 12:52:55,757 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 12:53:03,306 [MainThread] INFO EvalMetric - ARand: 0.7415211217619814
2023-02-03 12:53:03,351 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 12:53:10,914 [MainThread] INFO EvalMetric - ARand: 0.7227469511671067
2023-02-03 12:53:10,960 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 12:53:18,535 [MainThread] INFO EvalMetric - ARand: 0.7568166981381828
2023-02-03 12:53:18,580 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 12:53:24,237 [MainThread] INFO EvalMetric - ARand: 0.779188524997125
2023-02-03 12:53:24,623 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4825506922842442. Evaluation score: 0.7010407102050504
2023-02-03 12:53:24,644 [MainThread] INFO UNet3DTrainer - Saving new best evaluation metric: 0.7010407102050504
2023-02-03 12:53:24,645 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 12:53:32,215 [MainThread] INFO EvalMetric - ARand: 0.7693203696370577
2023-02-03 12:53:32,235 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.216220997646451. Evaluation score: 0.7693203696370577
2023-02-03 12:53:32,236 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 12:53:33,293 [MainThread] INFO UNet3DTrainer - Training iteration [601/60000]. Epoch [10/499]
2023-02-03 12:53:33,664 [MainThread] INFO UNet3DTrainer - Training iteration [602/60000]. Epoch [10/499]
2023-02-03 12:53:34,191 [MainThread] INFO UNet3DTrainer - Training iteration [603/60000]. Epoch [10/499]
2023-02-03 12:53:34,735 [MainThread] INFO UNet3DTrainer - Training iteration [604/60000]. Epoch [10/499]
2023-02-03 12:53:35,295 [MainThread] INFO UNet3DTrainer - Training iteration [605/60000]. Epoch [10/499]
2023-02-03 12:53:35,846 [MainThread] INFO UNet3DTrainer - Training iteration [606/60000]. Epoch [10/499]
2023-02-03 12:53:36,379 [MainThread] INFO UNet3DTrainer - Training iteration [607/60000]. Epoch [10/499]
2023-02-03 12:53:36,917 [MainThread] INFO UNet3DTrainer - Training iteration [608/60000]. Epoch [10/499]
2023-02-03 12:53:37,455 [MainThread] INFO UNet3DTrainer - Training iteration [609/60000]. Epoch [10/499]
2023-02-03 12:53:37,981 [MainThread] INFO UNet3DTrainer - Training iteration [610/60000]. Epoch [10/499]
2023-02-03 12:53:38,510 [MainThread] INFO UNet3DTrainer - Training iteration [611/60000]. Epoch [10/499]
2023-02-03 12:53:39,043 [MainThread] INFO UNet3DTrainer - Training iteration [612/60000]. Epoch [10/499]
2023-02-03 12:53:39,575 [MainThread] INFO UNet3DTrainer - Training iteration [613/60000]. Epoch [10/499]
2023-02-03 12:53:55,963 [MainThread] INFO UNet3DTrainer - Training iteration [614/60000]. Epoch [10/499]
2023-02-03 12:53:56,351 [MainThread] INFO UNet3DTrainer - Training iteration [615/60000]. Epoch [10/499]
2023-02-03 12:53:56,876 [MainThread] INFO UNet3DTrainer - Training iteration [616/60000]. Epoch [10/499]
2023-02-03 12:54:27,795 [MainThread] INFO UNet3DTrainer - Training iteration [617/60000]. Epoch [11/499]
2023-02-03 12:54:28,818 [MainThread] INFO UNet3DTrainer - Training iteration [618/60000]. Epoch [11/499]
2023-02-03 12:54:29,179 [MainThread] INFO UNet3DTrainer - Training iteration [619/60000]. Epoch [11/499]
2023-02-03 12:54:29,760 [MainThread] INFO UNet3DTrainer - Training iteration [620/60000]. Epoch [11/499]
2023-02-03 12:54:50,570 [MainThread] INFO UNet3DTrainer - Training iteration [621/60000]. Epoch [11/499]
2023-02-03 12:54:50,960 [MainThread] INFO UNet3DTrainer - Training iteration [622/60000]. Epoch [11/499]
2023-02-03 12:54:51,491 [MainThread] INFO UNet3DTrainer - Training iteration [623/60000]. Epoch [11/499]
2023-02-03 12:54:56,258 [MainThread] INFO UNet3DTrainer - Training iteration [624/60000]. Epoch [11/499]
2023-02-03 12:54:56,684 [MainThread] INFO UNet3DTrainer - Training iteration [625/60000]. Epoch [11/499]
2023-02-03 12:54:57,370 [MainThread] INFO UNet3DTrainer - Training iteration [626/60000]. Epoch [11/499]
2023-02-03 12:54:57,934 [MainThread] INFO UNet3DTrainer - Training iteration [627/60000]. Epoch [11/499]
2023-02-03 12:54:58,576 [MainThread] INFO UNet3DTrainer - Training iteration [628/60000]. Epoch [11/499]
2023-02-03 12:54:59,107 [MainThread] INFO UNet3DTrainer - Training iteration [629/60000]. Epoch [11/499]
2023-02-03 12:55:01,770 [MainThread] INFO UNet3DTrainer - Training iteration [630/60000]. Epoch [11/499]
2023-02-03 12:55:02,296 [MainThread] INFO UNet3DTrainer - Training iteration [631/60000]. Epoch [11/499]
2023-02-03 12:55:02,876 [MainThread] INFO UNet3DTrainer - Training iteration [632/60000]. Epoch [11/499]
2023-02-03 12:55:25,312 [MainThread] INFO UNet3DTrainer - Training iteration [633/60000]. Epoch [11/499]
2023-02-03 12:55:25,771 [MainThread] INFO UNet3DTrainer - Training iteration [634/60000]. Epoch [11/499]
2023-02-03 12:55:26,316 [MainThread] INFO UNet3DTrainer - Training iteration [635/60000]. Epoch [11/499]
2023-02-03 12:55:32,310 [MainThread] INFO UNet3DTrainer - Training iteration [636/60000]. Epoch [11/499]
2023-02-03 12:55:49,636 [MainThread] INFO UNet3DTrainer - Training iteration [637/60000]. Epoch [11/499]
2023-02-03 12:55:50,040 [MainThread] INFO UNet3DTrainer - Training iteration [638/60000]. Epoch [11/499]
2023-02-03 12:55:52,236 [MainThread] INFO UNet3DTrainer - Training iteration [639/60000]. Epoch [11/499]
2023-02-03 12:55:52,617 [MainThread] INFO UNet3DTrainer - Training iteration [640/60000]. Epoch [11/499]
2023-02-03 12:55:53,210 [MainThread] INFO UNet3DTrainer - Training iteration [641/60000]. Epoch [11/499]
2023-02-03 12:55:53,840 [MainThread] INFO UNet3DTrainer - Training iteration [642/60000]. Epoch [11/499]
2023-02-03 12:55:54,830 [MainThread] INFO UNet3DTrainer - Training iteration [643/60000]. Epoch [11/499]
2023-02-03 12:55:55,282 [MainThread] INFO UNet3DTrainer - Training iteration [644/60000]. Epoch [11/499]
2023-02-03 12:56:18,931 [MainThread] INFO UNet3DTrainer - Training iteration [645/60000]. Epoch [11/499]
2023-02-03 12:56:19,322 [MainThread] INFO UNet3DTrainer - Training iteration [646/60000]. Epoch [11/499]
2023-02-03 12:56:20,004 [MainThread] INFO UNet3DTrainer - Training iteration [647/60000]. Epoch [11/499]
2023-02-03 12:56:25,424 [MainThread] INFO UNet3DTrainer - Training iteration [648/60000]. Epoch [11/499]
2023-02-03 12:56:25,796 [MainThread] INFO UNet3DTrainer - Training iteration [649/60000]. Epoch [11/499]
2023-02-03 12:56:26,353 [MainThread] INFO UNet3DTrainer - Training iteration [650/60000]. Epoch [11/499]
2023-02-03 12:56:26,914 [MainThread] INFO UNet3DTrainer - Training iteration [651/60000]. Epoch [11/499]
2023-02-03 12:56:27,456 [MainThread] INFO UNet3DTrainer - Training iteration [652/60000]. Epoch [11/499]
2023-02-03 12:56:28,022 [MainThread] INFO UNet3DTrainer - Training iteration [653/60000]. Epoch [11/499]
2023-02-03 12:56:29,730 [MainThread] INFO UNet3DTrainer - Training iteration [654/60000]. Epoch [11/499]
2023-02-03 12:56:30,166 [MainThread] INFO UNet3DTrainer - Training iteration [655/60000]. Epoch [11/499]
2023-02-03 12:56:30,753 [MainThread] INFO UNet3DTrainer - Training iteration [656/60000]. Epoch [11/499]
2023-02-03 12:56:31,330 [MainThread] INFO UNet3DTrainer - Training iteration [657/60000]. Epoch [11/499]
2023-02-03 12:57:06,447 [MainThread] INFO UNet3DTrainer - Training iteration [658/60000]. Epoch [11/499]
2023-02-03 12:57:06,822 [MainThread] INFO UNet3DTrainer - Training iteration [659/60000]. Epoch [11/499]
2023-02-03 12:57:07,358 [MainThread] INFO UNet3DTrainer - Training iteration [660/60000]. Epoch [11/499]
2023-02-03 12:57:07,900 [MainThread] INFO UNet3DTrainer - Training iteration [661/60000]. Epoch [11/499]
2023-02-03 12:57:08,439 [MainThread] INFO UNet3DTrainer - Training iteration [662/60000]. Epoch [11/499]
2023-02-03 12:57:08,971 [MainThread] INFO UNet3DTrainer - Training iteration [663/60000]. Epoch [11/499]
2023-02-03 12:57:09,681 [MainThread] INFO UNet3DTrainer - Training iteration [664/60000]. Epoch [11/499]
2023-02-03 12:57:10,052 [MainThread] INFO UNet3DTrainer - Training iteration [665/60000]. Epoch [11/499]
2023-02-03 12:57:10,601 [MainThread] INFO UNet3DTrainer - Training iteration [666/60000]. Epoch [11/499]
2023-02-03 12:57:11,144 [MainThread] INFO UNet3DTrainer - Training iteration [667/60000]. Epoch [11/499]
2023-02-03 12:57:11,668 [MainThread] INFO UNet3DTrainer - Training iteration [668/60000]. Epoch [11/499]
2023-02-03 12:57:12,208 [MainThread] INFO UNet3DTrainer - Training iteration [669/60000]. Epoch [11/499]
2023-02-03 12:57:49,263 [MainThread] INFO UNet3DTrainer - Training iteration [670/60000]. Epoch [11/499]
2023-02-03 12:57:49,638 [MainThread] INFO UNet3DTrainer - Training iteration [671/60000]. Epoch [11/499]
2023-02-03 12:57:50,174 [MainThread] INFO UNet3DTrainer - Training iteration [672/60000]. Epoch [11/499]
2023-02-03 12:58:58,620 [MainThread] INFO UNet3DTrainer - Training iteration [673/60000]. Epoch [12/499]
2023-02-03 12:58:58,993 [MainThread] INFO UNet3DTrainer - Training iteration [674/60000]. Epoch [12/499]
2023-02-03 12:58:59,513 [MainThread] INFO UNet3DTrainer - Training iteration [675/60000]. Epoch [12/499]
2023-02-03 12:59:00,052 [MainThread] INFO UNet3DTrainer - Training iteration [676/60000]. Epoch [12/499]
2023-02-03 12:59:00,602 [MainThread] INFO UNet3DTrainer - Training iteration [677/60000]. Epoch [12/499]
2023-02-03 12:59:01,167 [MainThread] INFO UNet3DTrainer - Training iteration [678/60000]. Epoch [12/499]
2023-02-03 12:59:21,097 [MainThread] INFO UNet3DTrainer - Training iteration [679/60000]. Epoch [12/499]
2023-02-03 12:59:21,470 [MainThread] INFO UNet3DTrainer - Training iteration [680/60000]. Epoch [12/499]
2023-02-03 12:59:22,008 [MainThread] INFO UNet3DTrainer - Training iteration [681/60000]. Epoch [12/499]
2023-02-03 12:59:22,558 [MainThread] INFO UNet3DTrainer - Training iteration [682/60000]. Epoch [12/499]
2023-02-03 12:59:23,117 [MainThread] INFO UNet3DTrainer - Training iteration [683/60000]. Epoch [12/499]
2023-02-03 12:59:23,674 [MainThread] INFO UNet3DTrainer - Training iteration [684/60000]. Epoch [12/499]
2023-02-03 12:59:25,220 [MainThread] INFO UNet3DTrainer - Training iteration [685/60000]. Epoch [12/499]
2023-02-03 12:59:25,606 [MainThread] INFO UNet3DTrainer - Training iteration [686/60000]. Epoch [12/499]
2023-02-03 12:59:26,230 [MainThread] INFO UNet3DTrainer - Training iteration [687/60000]. Epoch [12/499]
2023-02-03 12:59:26,806 [MainThread] INFO UNet3DTrainer - Training iteration [688/60000]. Epoch [12/499]
2023-02-03 13:00:04,476 [MainThread] INFO UNet3DTrainer - Training iteration [689/60000]. Epoch [12/499]
2023-02-03 13:00:04,864 [MainThread] INFO UNet3DTrainer - Training iteration [690/60000]. Epoch [12/499]
2023-02-03 13:00:05,387 [MainThread] INFO UNet3DTrainer - Training iteration [691/60000]. Epoch [12/499]
2023-02-03 13:00:05,941 [MainThread] INFO UNet3DTrainer - Training iteration [692/60000]. Epoch [12/499]
2023-02-03 13:00:06,479 [MainThread] INFO UNet3DTrainer - Training iteration [693/60000]. Epoch [12/499]
2023-02-03 13:00:07,045 [MainThread] INFO UNet3DTrainer - Training iteration [694/60000]. Epoch [12/499]
2023-02-03 13:00:30,514 [MainThread] INFO UNet3DTrainer - Training iteration [695/60000]. Epoch [12/499]
2023-02-03 13:00:30,914 [MainThread] INFO UNet3DTrainer - Training iteration [696/60000]. Epoch [12/499]
2023-02-03 13:00:31,457 [MainThread] INFO UNet3DTrainer - Training iteration [697/60000]. Epoch [12/499]
2023-02-03 13:00:31,989 [MainThread] INFO UNet3DTrainer - Training iteration [698/60000]. Epoch [12/499]
2023-02-03 13:00:32,543 [MainThread] INFO UNet3DTrainer - Training iteration [699/60000]. Epoch [12/499]
2023-02-03 13:00:33,106 [MainThread] INFO UNet3DTrainer - Training iteration [700/60000]. Epoch [12/499]
2023-02-03 13:00:33,700 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 13:00:40,040 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 13:00:49,304 [MainThread] INFO EvalMetric - ARand: 0.6417258405253439
2023-02-03 13:00:49,340 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 13:00:56,748 [MainThread] INFO EvalMetric - ARand: 0.6641071651752036
2023-02-03 13:00:56,785 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 13:01:04,424 [MainThread] INFO EvalMetric - ARand: 0.7209864166787504
2023-02-03 13:01:04,459 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 13:01:11,803 [MainThread] INFO EvalMetric - ARand: 0.6139693173502738
2023-02-03 13:01:11,841 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 13:01:19,289 [MainThread] INFO EvalMetric - ARand: 0.6421570094383446
2023-02-03 13:01:19,335 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 13:01:26,723 [MainThread] INFO EvalMetric - ARand: 0.6964838106435179
2023-02-03 13:01:26,765 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 13:01:34,108 [MainThread] INFO EvalMetric - ARand: 0.5462847970093586
2023-02-03 13:01:34,148 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 13:01:41,522 [MainThread] INFO EvalMetric - ARand: 0.6347546842015968
2023-02-03 13:01:41,560 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 13:01:48,894 [MainThread] INFO EvalMetric - ARand: 0.64318574651352
2023-02-03 13:01:48,930 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 13:01:56,389 [MainThread] INFO EvalMetric - ARand: 0.6604328679088469
2023-02-03 13:01:56,434 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 13:02:03,916 [MainThread] INFO EvalMetric - ARand: 0.6016112467963612
2023-02-03 13:02:03,958 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 13:02:11,389 [MainThread] INFO EvalMetric - ARand: 0.5650972653743906
2023-02-03 13:02:11,430 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 13:02:18,860 [MainThread] INFO EvalMetric - ARand: 0.6745231796503955
2023-02-03 13:02:18,901 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 13:02:26,328 [MainThread] INFO EvalMetric - ARand: 0.6541057216825146
2023-02-03 13:02:26,372 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 13:02:33,863 [MainThread] INFO EvalMetric - ARand: 0.5892484775627043
2023-02-03 13:02:33,906 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 13:02:41,380 [MainThread] INFO EvalMetric - ARand: 0.6058375333307272
2023-02-03 13:02:41,423 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 13:02:48,918 [MainThread] INFO EvalMetric - ARand: 0.5978750849365461
2023-02-03 13:02:48,962 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 13:02:56,462 [MainThread] INFO EvalMetric - ARand: 0.651145834269731
2023-02-03 13:02:56,507 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 13:03:04,028 [MainThread] INFO EvalMetric - ARand: 0.6630596486157019
2023-02-03 13:03:04,071 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 13:03:11,599 [MainThread] INFO EvalMetric - ARand: 0.6559902384284955
2023-02-03 13:03:11,646 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 13:03:19,167 [MainThread] INFO EvalMetric - ARand: 0.7330546316512124
2023-02-03 13:03:19,212 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 13:03:24,878 [MainThread] INFO EvalMetric - ARand: 0.712953283991738
2023-02-03 13:03:25,264 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4818386250528797. Evaluation score: 0.6432345508384983
2023-02-03 13:03:25,287 [MainThread] INFO UNet3DTrainer - Saving new best evaluation metric: 0.6432345508384983
2023-02-03 13:03:25,289 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 13:03:32,772 [MainThread] INFO EvalMetric - ARand: 0.7483378678327726
2023-02-03 13:03:32,791 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.20082082386527741. Evaluation score: 0.7483378678327726
2023-02-03 13:03:32,792 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 13:03:33,863 [MainThread] INFO UNet3DTrainer - Training iteration [701/60000]. Epoch [12/499]
2023-02-03 13:03:34,234 [MainThread] INFO UNet3DTrainer - Training iteration [702/60000]. Epoch [12/499]
2023-02-03 13:03:34,765 [MainThread] INFO UNet3DTrainer - Training iteration [703/60000]. Epoch [12/499]
2023-02-03 13:03:35,314 [MainThread] INFO UNet3DTrainer - Training iteration [704/60000]. Epoch [12/499]
2023-02-03 13:03:35,862 [MainThread] INFO UNet3DTrainer - Training iteration [705/60000]. Epoch [12/499]
2023-02-03 13:03:36,409 [MainThread] INFO UNet3DTrainer - Training iteration [706/60000]. Epoch [12/499]
2023-02-03 13:03:37,020 [MainThread] INFO UNet3DTrainer - Training iteration [707/60000]. Epoch [12/499]
2023-02-03 13:03:37,561 [MainThread] INFO UNet3DTrainer - Training iteration [708/60000]. Epoch [12/499]
2023-02-03 13:03:38,098 [MainThread] INFO UNet3DTrainer - Training iteration [709/60000]. Epoch [12/499]
2023-02-03 13:03:38,669 [MainThread] INFO UNet3DTrainer - Training iteration [710/60000]. Epoch [12/499]
2023-02-03 13:03:39,220 [MainThread] INFO UNet3DTrainer - Training iteration [711/60000]. Epoch [12/499]
2023-02-03 13:03:39,782 [MainThread] INFO UNet3DTrainer - Training iteration [712/60000]. Epoch [12/499]
2023-02-03 13:04:18,898 [MainThread] INFO UNet3DTrainer - Training iteration [713/60000]. Epoch [12/499]
2023-02-03 13:04:20,614 [MainThread] INFO UNet3DTrainer - Training iteration [714/60000]. Epoch [12/499]
2023-02-03 13:04:20,953 [MainThread] INFO UNet3DTrainer - Training iteration [715/60000]. Epoch [12/499]
2023-02-03 13:04:21,502 [MainThread] INFO UNet3DTrainer - Training iteration [716/60000]. Epoch [12/499]
2023-02-03 13:04:22,047 [MainThread] INFO UNet3DTrainer - Training iteration [717/60000]. Epoch [12/499]
2023-02-03 13:04:22,609 [MainThread] INFO UNet3DTrainer - Training iteration [718/60000]. Epoch [12/499]
2023-02-03 13:04:41,103 [MainThread] INFO UNet3DTrainer - Training iteration [719/60000]. Epoch [12/499]
2023-02-03 13:05:00,961 [MainThread] INFO UNet3DTrainer - Training iteration [720/60000]. Epoch [12/499]
2023-02-03 13:05:01,329 [MainThread] INFO UNet3DTrainer - Training iteration [721/60000]. Epoch [12/499]
2023-02-03 13:05:01,857 [MainThread] INFO UNet3DTrainer - Training iteration [722/60000]. Epoch [12/499]
2023-02-03 13:05:02,392 [MainThread] INFO UNet3DTrainer - Training iteration [723/60000]. Epoch [12/499]
2023-02-03 13:05:02,921 [MainThread] INFO UNet3DTrainer - Training iteration [724/60000]. Epoch [12/499]
2023-02-03 13:05:03,455 [MainThread] INFO UNet3DTrainer - Training iteration [725/60000]. Epoch [12/499]
2023-02-03 13:05:03,999 [MainThread] INFO UNet3DTrainer - Training iteration [726/60000]. Epoch [12/499]
2023-02-03 13:05:04,511 [MainThread] INFO UNet3DTrainer - Training iteration [727/60000]. Epoch [12/499]
2023-02-03 13:05:05,040 [MainThread] INFO UNet3DTrainer - Training iteration [728/60000]. Epoch [12/499]
2023-02-03 13:05:36,799 [MainThread] INFO UNet3DTrainer - Training iteration [729/60000]. Epoch [13/499]
2023-02-03 13:05:37,227 [MainThread] INFO UNet3DTrainer - Training iteration [730/60000]. Epoch [13/499]
2023-02-03 13:05:37,830 [MainThread] INFO UNet3DTrainer - Training iteration [731/60000]. Epoch [13/499]
2023-02-03 13:05:38,473 [MainThread] INFO UNet3DTrainer - Training iteration [732/60000]. Epoch [13/499]
2023-02-03 13:05:39,040 [MainThread] INFO UNet3DTrainer - Training iteration [733/60000]. Epoch [13/499]
2023-02-03 13:05:39,626 [MainThread] INFO UNet3DTrainer - Training iteration [734/60000]. Epoch [13/499]
2023-02-03 13:06:00,828 [MainThread] INFO UNet3DTrainer - Training iteration [735/60000]. Epoch [13/499]
2023-02-03 13:06:01,210 [MainThread] INFO UNet3DTrainer - Training iteration [736/60000]. Epoch [13/499]
2023-02-03 13:06:01,756 [MainThread] INFO UNet3DTrainer - Training iteration [737/60000]. Epoch [13/499]
2023-02-03 13:06:19,675 [MainThread] INFO UNet3DTrainer - Training iteration [738/60000]. Epoch [13/499]
2023-02-03 13:06:20,069 [MainThread] INFO UNet3DTrainer - Training iteration [739/60000]. Epoch [13/499]
2023-02-03 13:06:20,604 [MainThread] INFO UNet3DTrainer - Training iteration [740/60000]. Epoch [13/499]
2023-02-03 13:06:21,146 [MainThread] INFO UNet3DTrainer - Training iteration [741/60000]. Epoch [13/499]
2023-02-03 13:06:21,682 [MainThread] INFO UNet3DTrainer - Training iteration [742/60000]. Epoch [13/499]
2023-02-03 13:06:22,251 [MainThread] INFO UNet3DTrainer - Training iteration [743/60000]. Epoch [13/499]
2023-02-03 13:06:24,066 [MainThread] INFO UNet3DTrainer - Training iteration [744/60000]. Epoch [13/499]
2023-02-03 13:06:24,480 [MainThread] INFO UNet3DTrainer - Training iteration [745/60000]. Epoch [13/499]
2023-02-03 13:06:24,997 [MainThread] INFO UNet3DTrainer - Training iteration [746/60000]. Epoch [13/499]
2023-02-03 13:06:29,808 [MainThread] INFO UNet3DTrainer - Training iteration [747/60000]. Epoch [13/499]
2023-02-03 13:06:30,195 [MainThread] INFO UNet3DTrainer - Training iteration [748/60000]. Epoch [13/499]
2023-02-03 13:06:30,740 [MainThread] INFO UNet3DTrainer - Training iteration [749/60000]. Epoch [13/499]
2023-02-03 13:06:31,286 [MainThread] INFO UNet3DTrainer - Training iteration [750/60000]. Epoch [13/499]
2023-02-03 13:06:31,850 [MainThread] INFO UNet3DTrainer - Training iteration [751/60000]. Epoch [13/499]
2023-02-03 13:06:32,434 [MainThread] INFO UNet3DTrainer - Training iteration [752/60000]. Epoch [13/499]
2023-02-03 13:06:34,451 [MainThread] INFO UNet3DTrainer - Training iteration [753/60000]. Epoch [13/499]
2023-02-03 13:06:35,046 [MainThread] INFO UNet3DTrainer - Training iteration [754/60000]. Epoch [13/499]
2023-02-03 13:06:35,610 [MainThread] INFO UNet3DTrainer - Training iteration [755/60000]. Epoch [13/499]
2023-02-03 13:07:13,530 [MainThread] INFO UNet3DTrainer - Training iteration [756/60000]. Epoch [13/499]
2023-02-03 13:07:13,904 [MainThread] INFO UNet3DTrainer - Training iteration [757/60000]. Epoch [13/499]
2023-02-03 13:07:14,438 [MainThread] INFO UNet3DTrainer - Training iteration [758/60000]. Epoch [13/499]
2023-02-03 13:07:14,977 [MainThread] INFO UNet3DTrainer - Training iteration [759/60000]. Epoch [13/499]
2023-02-03 13:07:15,529 [MainThread] INFO UNet3DTrainer - Training iteration [760/60000]. Epoch [13/499]
2023-02-03 13:07:16,086 [MainThread] INFO UNet3DTrainer - Training iteration [761/60000]. Epoch [13/499]
2023-02-03 13:07:42,196 [MainThread] INFO UNet3DTrainer - Training iteration [762/60000]. Epoch [13/499]
2023-02-03 13:07:42,595 [MainThread] INFO UNet3DTrainer - Training iteration [763/60000]. Epoch [13/499]
2023-02-03 13:07:43,155 [MainThread] INFO UNet3DTrainer - Training iteration [764/60000]. Epoch [13/499]
2023-02-03 13:07:43,686 [MainThread] INFO UNet3DTrainer - Training iteration [765/60000]. Epoch [13/499]
2023-02-03 13:07:44,219 [MainThread] INFO UNet3DTrainer - Training iteration [766/60000]. Epoch [13/499]
2023-02-03 13:07:44,770 [MainThread] INFO UNet3DTrainer - Training iteration [767/60000]. Epoch [13/499]
2023-02-03 13:07:47,432 [MainThread] INFO UNet3DTrainer - Training iteration [768/60000]. Epoch [13/499]
2023-02-03 13:07:47,942 [MainThread] INFO UNet3DTrainer - Training iteration [769/60000]. Epoch [13/499]
2023-02-03 13:08:08,738 [MainThread] INFO UNet3DTrainer - Training iteration [770/60000]. Epoch [13/499]
2023-02-03 13:08:09,127 [MainThread] INFO UNet3DTrainer - Training iteration [771/60000]. Epoch [13/499]
2023-02-03 13:08:09,692 [MainThread] INFO UNet3DTrainer - Training iteration [772/60000]. Epoch [13/499]
2023-02-03 13:08:10,265 [MainThread] INFO UNet3DTrainer - Training iteration [773/60000]. Epoch [13/499]
2023-02-03 13:08:37,138 [MainThread] INFO UNet3DTrainer - Training iteration [774/60000]. Epoch [13/499]
2023-02-03 13:08:37,522 [MainThread] INFO UNet3DTrainer - Training iteration [775/60000]. Epoch [13/499]
2023-02-03 13:08:38,039 [MainThread] INFO UNet3DTrainer - Training iteration [776/60000]. Epoch [13/499]
2023-02-03 13:08:38,578 [MainThread] INFO UNet3DTrainer - Training iteration [777/60000]. Epoch [13/499]
2023-02-03 13:08:39,106 [MainThread] INFO UNet3DTrainer - Training iteration [778/60000]. Epoch [13/499]
2023-02-03 13:08:39,636 [MainThread] INFO UNet3DTrainer - Training iteration [779/60000]. Epoch [13/499]
2023-02-03 13:08:58,459 [MainThread] INFO UNet3DTrainer - Training iteration [780/60000]. Epoch [13/499]
2023-02-03 13:08:58,835 [MainThread] INFO UNet3DTrainer - Training iteration [781/60000]. Epoch [13/499]
2023-02-03 13:08:59,359 [MainThread] INFO UNet3DTrainer - Training iteration [782/60000]. Epoch [13/499]
2023-02-03 13:08:59,884 [MainThread] INFO UNet3DTrainer - Training iteration [783/60000]. Epoch [13/499]
2023-02-03 13:09:00,428 [MainThread] INFO UNet3DTrainer - Training iteration [784/60000]. Epoch [13/499]
2023-02-03 13:09:08,230 [MainThread] INFO UNet3DTrainer - Training iteration [785/60000]. Epoch [14/499]
2023-02-03 13:09:33,960 [MainThread] INFO UNet3DTrainer - Training iteration [786/60000]. Epoch [14/499]
2023-02-03 13:09:34,410 [MainThread] INFO UNet3DTrainer - Training iteration [787/60000]. Epoch [14/499]
2023-02-03 13:09:56,110 [MainThread] INFO UNet3DTrainer - Training iteration [788/60000]. Epoch [14/499]
2023-02-03 13:09:56,493 [MainThread] INFO UNet3DTrainer - Training iteration [789/60000]. Epoch [14/499]
2023-02-03 13:09:57,050 [MainThread] INFO UNet3DTrainer - Training iteration [790/60000]. Epoch [14/499]
2023-02-03 13:10:00,246 [MainThread] INFO UNet3DTrainer - Training iteration [791/60000]. Epoch [14/499]
2023-02-03 13:10:00,652 [MainThread] INFO UNet3DTrainer - Training iteration [792/60000]. Epoch [14/499]
2023-02-03 13:10:01,211 [MainThread] INFO UNet3DTrainer - Training iteration [793/60000]. Epoch [14/499]
2023-02-03 13:10:21,935 [MainThread] INFO UNet3DTrainer - Training iteration [794/60000]. Epoch [14/499]
2023-02-03 13:10:22,305 [MainThread] INFO UNet3DTrainer - Training iteration [795/60000]. Epoch [14/499]
2023-02-03 13:10:22,858 [MainThread] INFO UNet3DTrainer - Training iteration [796/60000]. Epoch [14/499]
2023-02-03 13:10:23,418 [MainThread] INFO UNet3DTrainer - Training iteration [797/60000]. Epoch [14/499]
2023-02-03 13:10:28,160 [MainThread] INFO UNet3DTrainer - Training iteration [798/60000]. Epoch [14/499]
2023-02-03 13:10:28,546 [MainThread] INFO UNet3DTrainer - Training iteration [799/60000]. Epoch [14/499]
2023-02-03 13:11:06,090 [MainThread] INFO UNet3DTrainer - Training iteration [800/60000]. Epoch [14/499]
2023-02-03 13:11:06,440 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 13:11:12,285 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 13:11:21,999 [MainThread] INFO EvalMetric - ARand: 0.6640584513939349
2023-02-03 13:11:22,041 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 13:11:29,508 [MainThread] INFO EvalMetric - ARand: 0.6747266783073578
2023-02-03 13:11:29,549 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 13:11:36,973 [MainThread] INFO EvalMetric - ARand: 0.751473626082569
2023-02-03 13:11:37,009 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 13:11:44,440 [MainThread] INFO EvalMetric - ARand: 0.660396520624163
2023-02-03 13:11:44,482 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 13:11:51,761 [MainThread] INFO EvalMetric - ARand: 0.6997538689801754
2023-02-03 13:11:51,796 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 13:11:59,232 [MainThread] INFO EvalMetric - ARand: 0.7334563765017412
2023-02-03 13:11:59,270 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 13:12:06,628 [MainThread] INFO EvalMetric - ARand: 0.653230055828848
2023-02-03 13:12:06,671 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 13:12:13,994 [MainThread] INFO EvalMetric - ARand: 0.7212025733743406
2023-02-03 13:12:14,028 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 13:12:21,309 [MainThread] INFO EvalMetric - ARand: 0.7001426661788673
2023-02-03 13:12:21,346 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 13:12:28,636 [MainThread] INFO EvalMetric - ARand: 0.7141869995045957
2023-02-03 13:12:28,676 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 13:12:36,113 [MainThread] INFO EvalMetric - ARand: 0.6812097204266427
2023-02-03 13:12:36,150 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 13:12:43,533 [MainThread] INFO EvalMetric - ARand: 0.6496910978319945
2023-02-03 13:12:43,573 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 13:12:50,855 [MainThread] INFO EvalMetric - ARand: 0.6758316140207316
2023-02-03 13:12:50,896 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 13:12:58,270 [MainThread] INFO EvalMetric - ARand: 0.7481785770067048
2023-02-03 13:12:58,313 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 13:13:05,782 [MainThread] INFO EvalMetric - ARand: 0.6840500849259796
2023-02-03 13:13:05,825 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 13:13:13,296 [MainThread] INFO EvalMetric - ARand: 0.7192920027512133
2023-02-03 13:13:13,338 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 13:13:20,770 [MainThread] INFO EvalMetric - ARand: 0.7169991012893104
2023-02-03 13:13:20,813 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 13:13:28,356 [MainThread] INFO EvalMetric - ARand: 0.6960255801879087
2023-02-03 13:13:28,399 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 13:13:35,818 [MainThread] INFO EvalMetric - ARand: 0.7078408601903625
2023-02-03 13:13:35,860 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 13:13:43,347 [MainThread] INFO EvalMetric - ARand: 0.7060396777685563
2023-02-03 13:13:43,390 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 13:13:50,856 [MainThread] INFO EvalMetric - ARand: 0.7623474053694868
2023-02-03 13:13:50,898 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 13:13:56,640 [MainThread] INFO EvalMetric - ARand: 0.7770008517785967
2023-02-03 13:13:57,026 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4806946036459385. Evaluation score: 0.7035808817185947
2023-02-03 13:13:57,049 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 13:14:04,384 [MainThread] INFO EvalMetric - ARand: 0.6732509085145324
2023-02-03 13:14:04,404 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.2063482254743576. Evaluation score: 0.6732509085145324
2023-02-03 13:14:04,405 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 13:14:05,495 [MainThread] INFO UNet3DTrainer - Training iteration [801/60000]. Epoch [14/499]
2023-02-03 13:14:05,874 [MainThread] INFO UNet3DTrainer - Training iteration [802/60000]. Epoch [14/499]
2023-02-03 13:14:06,409 [MainThread] INFO UNet3DTrainer - Training iteration [803/60000]. Epoch [14/499]
2023-02-03 13:14:06,979 [MainThread] INFO UNet3DTrainer - Training iteration [804/60000]. Epoch [14/499]
2023-02-03 13:14:07,511 [MainThread] INFO UNet3DTrainer - Training iteration [805/60000]. Epoch [14/499]
2023-02-03 13:14:08,078 [MainThread] INFO UNet3DTrainer - Training iteration [806/60000]. Epoch [14/499]
2023-02-03 13:14:08,630 [MainThread] INFO UNet3DTrainer - Training iteration [807/60000]. Epoch [14/499]
2023-02-03 13:14:09,233 [MainThread] INFO UNet3DTrainer - Training iteration [808/60000]. Epoch [14/499]
2023-02-03 13:14:09,893 [MainThread] INFO UNet3DTrainer - Training iteration [809/60000]. Epoch [14/499]
2023-02-03 13:14:10,450 [MainThread] INFO UNet3DTrainer - Training iteration [810/60000]. Epoch [14/499]
2023-02-03 13:14:11,090 [MainThread] INFO UNet3DTrainer - Training iteration [811/60000]. Epoch [14/499]
2023-02-03 13:14:11,680 [MainThread] INFO UNet3DTrainer - Training iteration [812/60000]. Epoch [14/499]
2023-02-03 13:14:12,269 [MainThread] INFO UNet3DTrainer - Training iteration [813/60000]. Epoch [14/499]
2023-02-03 13:14:12,874 [MainThread] INFO UNet3DTrainer - Training iteration [814/60000]. Epoch [14/499]
2023-02-03 13:14:13,450 [MainThread] INFO UNet3DTrainer - Training iteration [815/60000]. Epoch [14/499]
2023-02-03 13:14:13,982 [MainThread] INFO UNet3DTrainer - Training iteration [816/60000]. Epoch [14/499]
2023-02-03 13:14:36,662 [MainThread] INFO UNet3DTrainer - Training iteration [817/60000]. Epoch [14/499]
2023-02-03 13:14:37,059 [MainThread] INFO UNet3DTrainer - Training iteration [818/60000]. Epoch [14/499]
2023-02-03 13:14:37,623 [MainThread] INFO UNet3DTrainer - Training iteration [819/60000]. Epoch [14/499]
2023-02-03 13:14:40,660 [MainThread] INFO UNet3DTrainer - Training iteration [820/60000]. Epoch [14/499]
2023-02-03 13:14:41,142 [MainThread] INFO UNet3DTrainer - Training iteration [821/60000]. Epoch [14/499]
2023-02-03 13:14:41,760 [MainThread] INFO UNet3DTrainer - Training iteration [822/60000]. Epoch [14/499]
2023-02-03 13:14:42,310 [MainThread] INFO UNet3DTrainer - Training iteration [823/60000]. Epoch [14/499]
2023-02-03 13:15:02,595 [MainThread] INFO UNet3DTrainer - Training iteration [824/60000]. Epoch [14/499]
2023-02-03 13:15:02,964 [MainThread] INFO UNet3DTrainer - Training iteration [825/60000]. Epoch [14/499]
2023-02-03 13:15:03,510 [MainThread] INFO UNet3DTrainer - Training iteration [826/60000]. Epoch [14/499]
2023-02-03 13:15:04,056 [MainThread] INFO UNet3DTrainer - Training iteration [827/60000]. Epoch [14/499]
2023-02-03 13:15:04,625 [MainThread] INFO UNet3DTrainer - Training iteration [828/60000]. Epoch [14/499]
2023-02-03 13:15:06,193 [MainThread] INFO UNet3DTrainer - Training iteration [829/60000]. Epoch [14/499]
2023-02-03 13:15:06,814 [MainThread] INFO UNet3DTrainer - Training iteration [830/60000]. Epoch [14/499]
2023-02-03 13:15:07,223 [MainThread] INFO UNet3DTrainer - Training iteration [831/60000]. Epoch [14/499]
2023-02-03 13:15:07,784 [MainThread] INFO UNet3DTrainer - Training iteration [832/60000]. Epoch [14/499]
2023-02-03 13:15:08,339 [MainThread] INFO UNet3DTrainer - Training iteration [833/60000]. Epoch [14/499]
2023-02-03 13:15:08,893 [MainThread] INFO UNet3DTrainer - Training iteration [834/60000]. Epoch [14/499]
2023-02-03 13:15:10,678 [MainThread] INFO UNet3DTrainer - Training iteration [835/60000]. Epoch [14/499]
2023-02-03 13:16:05,005 [MainThread] INFO UNet3DTrainer - Training iteration [836/60000]. Epoch [14/499]
2023-02-03 13:16:05,365 [MainThread] INFO UNet3DTrainer - Training iteration [837/60000]. Epoch [14/499]
2023-02-03 13:16:05,897 [MainThread] INFO UNet3DTrainer - Training iteration [838/60000]. Epoch [14/499]
2023-02-03 13:16:06,423 [MainThread] INFO UNet3DTrainer - Training iteration [839/60000]. Epoch [14/499]
2023-02-03 13:16:06,963 [MainThread] INFO UNet3DTrainer - Training iteration [840/60000]. Epoch [14/499]
2023-02-03 13:16:14,662 [MainThread] INFO UNet3DTrainer - Training iteration [841/60000]. Epoch [15/499]
2023-02-03 13:16:15,140 [MainThread] INFO UNet3DTrainer - Training iteration [842/60000]. Epoch [15/499]
2023-02-03 13:16:37,895 [MainThread] INFO UNet3DTrainer - Training iteration [843/60000]. Epoch [15/499]
2023-02-03 13:16:38,315 [MainThread] INFO UNet3DTrainer - Training iteration [844/60000]. Epoch [15/499]
2023-02-03 13:16:38,860 [MainThread] INFO UNet3DTrainer - Training iteration [845/60000]. Epoch [15/499]
2023-02-03 13:16:39,435 [MainThread] INFO UNet3DTrainer - Training iteration [846/60000]. Epoch [15/499]
2023-02-03 13:16:39,995 [MainThread] INFO UNet3DTrainer - Training iteration [847/60000]. Epoch [15/499]
2023-02-03 13:16:42,981 [MainThread] INFO UNet3DTrainer - Training iteration [848/60000]. Epoch [15/499]
2023-02-03 13:17:05,919 [MainThread] INFO UNet3DTrainer - Training iteration [849/60000]. Epoch [15/499]
2023-02-03 13:17:06,627 [MainThread] INFO UNet3DTrainer - Training iteration [850/60000]. Epoch [15/499]
2023-02-03 13:17:06,976 [MainThread] INFO UNet3DTrainer - Training iteration [851/60000]. Epoch [15/499]
2023-02-03 13:17:07,514 [MainThread] INFO UNet3DTrainer - Training iteration [852/60000]. Epoch [15/499]
2023-02-03 13:17:08,079 [MainThread] INFO UNet3DTrainer - Training iteration [853/60000]. Epoch [15/499]
2023-02-03 13:17:08,620 [MainThread] INFO UNet3DTrainer - Training iteration [854/60000]. Epoch [15/499]
2023-02-03 13:17:29,568 [MainThread] INFO UNet3DTrainer - Training iteration [855/60000]. Epoch [15/499]
2023-02-03 13:17:29,948 [MainThread] INFO UNet3DTrainer - Training iteration [856/60000]. Epoch [15/499]
2023-02-03 13:17:30,498 [MainThread] INFO UNet3DTrainer - Training iteration [857/60000]. Epoch [15/499]
2023-02-03 13:17:31,065 [MainThread] INFO UNet3DTrainer - Training iteration [858/60000]. Epoch [15/499]
2023-02-03 13:17:31,620 [MainThread] INFO UNet3DTrainer - Training iteration [859/60000]. Epoch [15/499]
2023-02-03 13:17:32,187 [MainThread] INFO UNet3DTrainer - Training iteration [860/60000]. Epoch [15/499]
2023-02-03 13:17:57,998 [MainThread] INFO UNet3DTrainer - Training iteration [861/60000]. Epoch [15/499]
2023-02-03 13:17:58,402 [MainThread] INFO UNet3DTrainer - Training iteration [862/60000]. Epoch [15/499]
2023-02-03 13:17:59,035 [MainThread] INFO UNet3DTrainer - Training iteration [863/60000]. Epoch [15/499]
2023-02-03 13:17:59,690 [MainThread] INFO UNet3DTrainer - Training iteration [864/60000]. Epoch [15/499]
2023-02-03 13:18:00,245 [MainThread] INFO UNet3DTrainer - Training iteration [865/60000]. Epoch [15/499]
2023-02-03 13:18:00,831 [MainThread] INFO UNet3DTrainer - Training iteration [866/60000]. Epoch [15/499]
2023-02-03 13:18:27,615 [MainThread] INFO UNet3DTrainer - Training iteration [867/60000]. Epoch [15/499]
2023-02-03 13:18:27,996 [MainThread] INFO UNet3DTrainer - Training iteration [868/60000]. Epoch [15/499]
2023-02-03 13:18:28,559 [MainThread] INFO UNet3DTrainer - Training iteration [869/60000]. Epoch [15/499]
2023-02-03 13:18:29,098 [MainThread] INFO UNet3DTrainer - Training iteration [870/60000]. Epoch [15/499]
2023-02-03 13:18:29,659 [MainThread] INFO UNet3DTrainer - Training iteration [871/60000]. Epoch [15/499]
2023-02-03 13:18:30,361 [MainThread] INFO UNet3DTrainer - Training iteration [872/60000]. Epoch [15/499]
2023-02-03 13:19:08,435 [MainThread] INFO UNet3DTrainer - Training iteration [873/60000]. Epoch [15/499]
2023-02-03 13:19:08,815 [MainThread] INFO UNet3DTrainer - Training iteration [874/60000]. Epoch [15/499]
2023-02-03 13:19:09,343 [MainThread] INFO UNet3DTrainer - Training iteration [875/60000]. Epoch [15/499]
2023-02-03 13:19:09,890 [MainThread] INFO UNet3DTrainer - Training iteration [876/60000]. Epoch [15/499]
2023-02-03 13:19:10,428 [MainThread] INFO UNet3DTrainer - Training iteration [877/60000]. Epoch [15/499]
2023-02-03 13:19:11,030 [MainThread] INFO UNet3DTrainer - Training iteration [878/60000]. Epoch [15/499]
2023-02-03 13:19:12,320 [MainThread] INFO UNet3DTrainer - Training iteration [879/60000]. Epoch [15/499]
2023-02-03 13:19:12,792 [MainThread] INFO UNet3DTrainer - Training iteration [880/60000]. Epoch [15/499]
2023-02-03 13:19:13,427 [MainThread] INFO UNet3DTrainer - Training iteration [881/60000]. Epoch [15/499]
2023-02-03 13:19:14,047 [MainThread] INFO UNet3DTrainer - Training iteration [882/60000]. Epoch [15/499]
2023-02-03 13:19:14,615 [MainThread] INFO UNet3DTrainer - Training iteration [883/60000]. Epoch [15/499]
2023-02-03 13:19:15,250 [MainThread] INFO UNet3DTrainer - Training iteration [884/60000]. Epoch [15/499]
2023-02-03 13:19:41,439 [MainThread] INFO UNet3DTrainer - Training iteration [885/60000]. Epoch [15/499]
2023-02-03 13:19:41,842 [MainThread] INFO UNet3DTrainer - Training iteration [886/60000]. Epoch [15/499]
2023-02-03 13:19:42,402 [MainThread] INFO UNet3DTrainer - Training iteration [887/60000]. Epoch [15/499]
2023-02-03 13:19:42,955 [MainThread] INFO UNet3DTrainer - Training iteration [888/60000]. Epoch [15/499]
2023-02-03 13:19:43,494 [MainThread] INFO UNet3DTrainer - Training iteration [889/60000]. Epoch [15/499]
2023-02-03 13:19:44,040 [MainThread] INFO UNet3DTrainer - Training iteration [890/60000]. Epoch [15/499]
2023-02-03 13:19:45,260 [MainThread] INFO UNet3DTrainer - Training iteration [891/60000]. Epoch [15/499]
2023-02-03 13:19:45,591 [MainThread] INFO UNet3DTrainer - Training iteration [892/60000]. Epoch [15/499]
2023-02-03 13:19:46,125 [MainThread] INFO UNet3DTrainer - Training iteration [893/60000]. Epoch [15/499]
2023-02-03 13:19:46,649 [MainThread] INFO UNet3DTrainer - Training iteration [894/60000]. Epoch [15/499]
2023-02-03 13:20:01,410 [MainThread] INFO UNet3DTrainer - Training iteration [895/60000]. Epoch [15/499]
2023-02-03 13:20:01,780 [MainThread] INFO UNet3DTrainer - Training iteration [896/60000]. Epoch [15/499]
2023-02-03 13:20:33,499 [MainThread] INFO UNet3DTrainer - Training iteration [897/60000]. Epoch [16/499]
2023-02-03 13:20:33,905 [MainThread] INFO UNet3DTrainer - Training iteration [898/60000]. Epoch [16/499]
2023-02-03 13:20:34,457 [MainThread] INFO UNet3DTrainer - Training iteration [899/60000]. Epoch [16/499]
2023-02-03 13:20:35,014 [MainThread] INFO UNet3DTrainer - Training iteration [900/60000]. Epoch [16/499]
2023-02-03 13:20:35,551 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 13:20:42,442 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 13:20:52,986 [MainThread] INFO EvalMetric - ARand: 0.6620105343509077
2023-02-03 13:20:53,034 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 13:21:01,598 [MainThread] INFO EvalMetric - ARand: 0.6902094141793265
2023-02-03 13:21:01,650 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 13:21:09,675 [MainThread] INFO EvalMetric - ARand: 0.7354831021571384
2023-02-03 13:21:09,721 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 13:21:17,145 [MainThread] INFO EvalMetric - ARand: 0.6727834769553279
2023-02-03 13:21:17,188 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 13:21:24,652 [MainThread] INFO EvalMetric - ARand: 0.6804296567244696
2023-02-03 13:21:24,694 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 13:21:32,097 [MainThread] INFO EvalMetric - ARand: 0.7523023571030625
2023-02-03 13:21:32,152 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 13:21:39,624 [MainThread] INFO EvalMetric - ARand: 0.6447988737353404
2023-02-03 13:21:39,669 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 13:21:47,040 [MainThread] INFO EvalMetric - ARand: 0.7276484459638184
2023-02-03 13:21:47,077 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 13:21:54,523 [MainThread] INFO EvalMetric - ARand: 0.6906983338215251
2023-02-03 13:21:54,564 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 13:22:02,115 [MainThread] INFO EvalMetric - ARand: 0.7009681111913837
2023-02-03 13:22:02,150 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 13:22:09,509 [MainThread] INFO EvalMetric - ARand: 0.6363135733817262
2023-02-03 13:22:09,547 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 13:22:16,922 [MainThread] INFO EvalMetric - ARand: 0.6419590099084858
2023-02-03 13:22:16,964 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 13:22:24,319 [MainThread] INFO EvalMetric - ARand: 0.711585407404657
2023-02-03 13:22:24,362 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 13:22:31,777 [MainThread] INFO EvalMetric - ARand: 0.7236005200350923
2023-02-03 13:22:31,825 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 13:22:39,303 [MainThread] INFO EvalMetric - ARand: 0.7057642007316585
2023-02-03 13:22:39,344 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 13:22:46,766 [MainThread] INFO EvalMetric - ARand: 0.6986367799320893
2023-02-03 13:22:46,807 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 13:22:54,330 [MainThread] INFO EvalMetric - ARand: 0.676068493652392
2023-02-03 13:22:54,373 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 13:23:01,929 [MainThread] INFO EvalMetric - ARand: 0.6898664982805557
2023-02-03 13:23:01,972 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 13:23:09,510 [MainThread] INFO EvalMetric - ARand: 0.6939690733841811
2023-02-03 13:23:09,555 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 13:23:17,113 [MainThread] INFO EvalMetric - ARand: 0.708965174898528
2023-02-03 13:23:17,158 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 13:23:24,710 [MainThread] INFO EvalMetric - ARand: 0.7345883736460108
2023-02-03 13:23:24,755 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 13:23:30,451 [MainThread] INFO EvalMetric - ARand: 0.7376015559081596
2023-02-03 13:23:30,844 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4817131195945301. Evaluation score: 0.6957172679709792
2023-02-03 13:23:30,867 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 13:23:38,162 [MainThread] INFO EvalMetric - ARand: 0.731175960980606
2023-02-03 13:23:38,181 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.2124868407845497. Evaluation score: 0.731175960980606
2023-02-03 13:23:38,182 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 13:23:39,231 [MainThread] INFO UNet3DTrainer - Training iteration [901/60000]. Epoch [16/499]
2023-02-03 13:23:39,602 [MainThread] INFO UNet3DTrainer - Training iteration [902/60000]. Epoch [16/499]
2023-02-03 13:23:40,140 [MainThread] INFO UNet3DTrainer - Training iteration [903/60000]. Epoch [16/499]
2023-02-03 13:23:40,662 [MainThread] INFO UNet3DTrainer - Training iteration [904/60000]. Epoch [16/499]
2023-02-03 13:23:41,223 [MainThread] INFO UNet3DTrainer - Training iteration [905/60000]. Epoch [16/499]
2023-02-03 13:23:41,820 [MainThread] INFO UNet3DTrainer - Training iteration [906/60000]. Epoch [16/499]
2023-02-03 13:23:42,452 [MainThread] INFO UNet3DTrainer - Training iteration [907/60000]. Epoch [16/499]
2023-02-03 13:23:43,049 [MainThread] INFO UNet3DTrainer - Training iteration [908/60000]. Epoch [16/499]
2023-02-03 13:23:43,740 [MainThread] INFO UNet3DTrainer - Training iteration [909/60000]. Epoch [16/499]
2023-02-03 13:23:44,370 [MainThread] INFO UNet3DTrainer - Training iteration [910/60000]. Epoch [16/499]
2023-02-03 13:23:44,964 [MainThread] INFO UNet3DTrainer - Training iteration [911/60000]. Epoch [16/499]
2023-02-03 13:23:45,580 [MainThread] INFO UNet3DTrainer - Training iteration [912/60000]. Epoch [16/499]
2023-02-03 13:23:46,161 [MainThread] INFO UNet3DTrainer - Training iteration [913/60000]. Epoch [16/499]
2023-02-03 13:24:06,603 [MainThread] INFO UNet3DTrainer - Training iteration [914/60000]. Epoch [16/499]
2023-02-03 13:24:06,989 [MainThread] INFO UNet3DTrainer - Training iteration [915/60000]. Epoch [16/499]
2023-02-03 13:24:07,536 [MainThread] INFO UNet3DTrainer - Training iteration [916/60000]. Epoch [16/499]
2023-02-03 13:24:08,087 [MainThread] INFO UNet3DTrainer - Training iteration [917/60000]. Epoch [16/499]
2023-02-03 13:24:08,647 [MainThread] INFO UNet3DTrainer - Training iteration [918/60000]. Epoch [16/499]
2023-02-03 13:24:09,209 [MainThread] INFO UNet3DTrainer - Training iteration [919/60000]. Epoch [16/499]
2023-02-03 13:24:32,311 [MainThread] INFO UNet3DTrainer - Training iteration [920/60000]. Epoch [16/499]
2023-02-03 13:24:32,706 [MainThread] INFO UNet3DTrainer - Training iteration [921/60000]. Epoch [16/499]
2023-02-03 13:24:33,280 [MainThread] INFO UNet3DTrainer - Training iteration [922/60000]. Epoch [16/499]
2023-02-03 13:24:35,185 [MainThread] INFO UNet3DTrainer - Training iteration [923/60000]. Epoch [16/499]
2023-02-03 13:24:56,288 [MainThread] INFO UNet3DTrainer - Training iteration [924/60000]. Epoch [16/499]
2023-02-03 13:24:56,669 [MainThread] INFO UNet3DTrainer - Training iteration [925/60000]. Epoch [16/499]
2023-02-03 13:24:57,199 [MainThread] INFO UNet3DTrainer - Training iteration [926/60000]. Epoch [16/499]
2023-02-03 13:24:57,764 [MainThread] INFO UNet3DTrainer - Training iteration [927/60000]. Epoch [16/499]
2023-02-03 13:24:58,308 [MainThread] INFO UNet3DTrainer - Training iteration [928/60000]. Epoch [16/499]
2023-02-03 13:24:58,930 [MainThread] INFO UNet3DTrainer - Training iteration [929/60000]. Epoch [16/499]
2023-02-03 13:25:22,002 [MainThread] INFO UNet3DTrainer - Training iteration [930/60000]. Epoch [16/499]
2023-02-03 13:25:22,401 [MainThread] INFO UNet3DTrainer - Training iteration [931/60000]. Epoch [16/499]
2023-02-03 13:25:22,960 [MainThread] INFO UNet3DTrainer - Training iteration [932/60000]. Epoch [16/499]
2023-02-03 13:25:23,511 [MainThread] INFO UNet3DTrainer - Training iteration [933/60000]. Epoch [16/499]
2023-02-03 13:25:24,099 [MainThread] INFO UNet3DTrainer - Training iteration [934/60000]. Epoch [16/499]
2023-02-03 13:25:24,782 [MainThread] INFO UNet3DTrainer - Training iteration [935/60000]. Epoch [16/499]
2023-02-03 13:25:26,800 [MainThread] INFO UNet3DTrainer - Training iteration [936/60000]. Epoch [16/499]
2023-02-03 13:25:27,290 [MainThread] INFO UNet3DTrainer - Training iteration [937/60000]. Epoch [16/499]
2023-02-03 13:25:27,907 [MainThread] INFO UNet3DTrainer - Training iteration [938/60000]. Epoch [16/499]
2023-02-03 13:25:28,473 [MainThread] INFO UNet3DTrainer - Training iteration [939/60000]. Epoch [16/499]
2023-02-03 13:25:29,090 [MainThread] INFO UNet3DTrainer - Training iteration [940/60000]. Epoch [16/499]
2023-02-03 13:25:29,655 [MainThread] INFO UNet3DTrainer - Training iteration [941/60000]. Epoch [16/499]
2023-02-03 13:25:31,737 [MainThread] INFO UNet3DTrainer - Training iteration [942/60000]. Epoch [16/499]
2023-02-03 13:26:09,027 [MainThread] INFO UNet3DTrainer - Training iteration [943/60000]. Epoch [16/499]
2023-02-03 13:26:09,400 [MainThread] INFO UNet3DTrainer - Training iteration [944/60000]. Epoch [16/499]
2023-02-03 13:26:09,934 [MainThread] INFO UNet3DTrainer - Training iteration [945/60000]. Epoch [16/499]
2023-02-03 13:26:10,470 [MainThread] INFO UNet3DTrainer - Training iteration [946/60000]. Epoch [16/499]
2023-02-03 13:26:10,998 [MainThread] INFO UNet3DTrainer - Training iteration [947/60000]. Epoch [16/499]
2023-02-03 13:26:11,534 [MainThread] INFO UNet3DTrainer - Training iteration [948/60000]. Epoch [16/499]
2023-02-03 13:27:06,673 [MainThread] INFO UNet3DTrainer - Training iteration [949/60000]. Epoch [16/499]
2023-02-03 13:27:07,047 [MainThread] INFO UNet3DTrainer - Training iteration [950/60000]. Epoch [16/499]
2023-02-03 13:27:07,578 [MainThread] INFO UNet3DTrainer - Training iteration [951/60000]. Epoch [16/499]
2023-02-03 13:27:08,106 [MainThread] INFO UNet3DTrainer - Training iteration [952/60000]. Epoch [16/499]
2023-02-03 13:27:15,720 [MainThread] INFO UNet3DTrainer - Training iteration [953/60000]. Epoch [17/499]
2023-02-03 13:27:37,437 [MainThread] INFO UNet3DTrainer - Training iteration [954/60000]. Epoch [17/499]
2023-02-03 13:27:37,832 [MainThread] INFO UNet3DTrainer - Training iteration [955/60000]. Epoch [17/499]
2023-02-03 13:27:38,384 [MainThread] INFO UNet3DTrainer - Training iteration [956/60000]. Epoch [17/499]
2023-02-03 13:27:39,391 [MainThread] INFO UNet3DTrainer - Training iteration [957/60000]. Epoch [17/499]
2023-02-03 13:27:39,755 [MainThread] INFO UNet3DTrainer - Training iteration [958/60000]. Epoch [17/499]
2023-02-03 13:27:40,313 [MainThread] INFO UNet3DTrainer - Training iteration [959/60000]. Epoch [17/499]
2023-02-03 13:28:19,195 [MainThread] INFO UNet3DTrainer - Training iteration [960/60000]. Epoch [17/499]
2023-02-03 13:28:19,577 [MainThread] INFO UNet3DTrainer - Training iteration [961/60000]. Epoch [17/499]
2023-02-03 13:28:20,103 [MainThread] INFO UNet3DTrainer - Training iteration [962/60000]. Epoch [17/499]
2023-02-03 13:28:20,670 [MainThread] INFO UNet3DTrainer - Training iteration [963/60000]. Epoch [17/499]
2023-02-03 13:28:21,205 [MainThread] INFO UNet3DTrainer - Training iteration [964/60000]. Epoch [17/499]
2023-02-03 13:28:21,767 [MainThread] INFO UNet3DTrainer - Training iteration [965/60000]. Epoch [17/499]
2023-02-03 13:28:43,663 [MainThread] INFO UNet3DTrainer - Training iteration [966/60000]. Epoch [17/499]
2023-02-03 13:28:44,040 [MainThread] INFO UNet3DTrainer - Training iteration [967/60000]. Epoch [17/499]
2023-02-03 13:28:44,601 [MainThread] INFO UNet3DTrainer - Training iteration [968/60000]. Epoch [17/499]
2023-02-03 13:28:45,158 [MainThread] INFO UNet3DTrainer - Training iteration [969/60000]. Epoch [17/499]
2023-02-03 13:28:45,724 [MainThread] INFO UNet3DTrainer - Training iteration [970/60000]. Epoch [17/499]
2023-02-03 13:28:46,304 [MainThread] INFO UNet3DTrainer - Training iteration [971/60000]. Epoch [17/499]
2023-02-03 13:28:48,898 [MainThread] INFO UNet3DTrainer - Training iteration [972/60000]. Epoch [17/499]
2023-02-03 13:28:49,323 [MainThread] INFO UNet3DTrainer - Training iteration [973/60000]. Epoch [17/499]
2023-02-03 13:28:49,874 [MainThread] INFO UNet3DTrainer - Training iteration [974/60000]. Epoch [17/499]
2023-02-03 13:28:50,484 [MainThread] INFO UNet3DTrainer - Training iteration [975/60000]. Epoch [17/499]
2023-02-03 13:28:51,124 [MainThread] INFO UNet3DTrainer - Training iteration [976/60000]. Epoch [17/499]
2023-02-03 13:28:51,724 [MainThread] INFO UNet3DTrainer - Training iteration [977/60000]. Epoch [17/499]
2023-02-03 13:28:53,462 [MainThread] INFO UNet3DTrainer - Training iteration [978/60000]. Epoch [17/499]
2023-02-03 13:28:53,873 [MainThread] INFO UNet3DTrainer - Training iteration [979/60000]. Epoch [17/499]
2023-02-03 13:28:54,450 [MainThread] INFO UNet3DTrainer - Training iteration [980/60000]. Epoch [17/499]
2023-02-03 13:29:19,511 [MainThread] INFO UNet3DTrainer - Training iteration [981/60000]. Epoch [17/499]
2023-02-03 13:29:44,047 [MainThread] INFO UNet3DTrainer - Training iteration [982/60000]. Epoch [17/499]
2023-02-03 13:29:44,500 [MainThread] INFO UNet3DTrainer - Training iteration [983/60000]. Epoch [17/499]
2023-02-03 13:29:45,120 [MainThread] INFO UNet3DTrainer - Training iteration [984/60000]. Epoch [17/499]
2023-02-03 13:29:45,914 [MainThread] INFO UNet3DTrainer - Training iteration [985/60000]. Epoch [17/499]
2023-02-03 13:29:46,410 [MainThread] INFO UNet3DTrainer - Training iteration [986/60000]. Epoch [17/499]
2023-02-03 13:29:49,916 [MainThread] INFO UNet3DTrainer - Training iteration [987/60000]. Epoch [17/499]
2023-02-03 13:29:50,418 [MainThread] INFO UNet3DTrainer - Training iteration [988/60000]. Epoch [17/499]
2023-02-03 13:30:11,008 [MainThread] INFO UNet3DTrainer - Training iteration [989/60000]. Epoch [17/499]
2023-02-03 13:30:11,409 [MainThread] INFO UNet3DTrainer - Training iteration [990/60000]. Epoch [17/499]
2023-02-03 13:30:11,947 [MainThread] INFO UNet3DTrainer - Training iteration [991/60000]. Epoch [17/499]
2023-02-03 13:30:12,500 [MainThread] INFO UNet3DTrainer - Training iteration [992/60000]. Epoch [17/499]
2023-02-03 13:30:15,560 [MainThread] INFO UNet3DTrainer - Training iteration [993/60000]. Epoch [17/499]
2023-02-03 13:30:15,988 [MainThread] INFO UNet3DTrainer - Training iteration [994/60000]. Epoch [17/499]
2023-02-03 13:30:16,572 [MainThread] INFO UNet3DTrainer - Training iteration [995/60000]. Epoch [17/499]
2023-02-03 13:30:17,228 [MainThread] INFO UNet3DTrainer - Training iteration [996/60000]. Epoch [17/499]
2023-02-03 13:30:35,161 [MainThread] INFO UNet3DTrainer - Training iteration [997/60000]. Epoch [17/499]
2023-02-03 13:30:35,546 [MainThread] INFO UNet3DTrainer - Training iteration [998/60000]. Epoch [17/499]
2023-02-03 13:30:36,087 [MainThread] INFO UNet3DTrainer - Training iteration [999/60000]. Epoch [17/499]
2023-02-03 13:30:36,628 [MainThread] INFO UNet3DTrainer - Training iteration [1000/60000]. Epoch [17/499]
2023-02-03 13:30:37,138 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 13:30:42,642 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 13:30:51,551 [MainThread] INFO EvalMetric - ARand: 0.6788135053693136
2023-02-03 13:30:51,599 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 13:30:59,127 [MainThread] INFO EvalMetric - ARand: 0.6848170430139562
2023-02-03 13:30:59,166 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 13:31:06,736 [MainThread] INFO EvalMetric - ARand: 0.7264806400495836
2023-02-03 13:31:06,784 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 13:31:14,217 [MainThread] INFO EvalMetric - ARand: 0.7041710654892137
2023-02-03 13:31:14,257 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 13:31:21,670 [MainThread] INFO EvalMetric - ARand: 0.7008505391276516
2023-02-03 13:31:21,711 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 13:31:29,132 [MainThread] INFO EvalMetric - ARand: 0.7361836156045072
2023-02-03 13:31:29,179 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 13:31:36,671 [MainThread] INFO EvalMetric - ARand: 0.6612773201450282
2023-02-03 13:31:36,717 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 13:31:44,082 [MainThread] INFO EvalMetric - ARand: 0.7210321353154745
2023-02-03 13:31:44,120 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 13:31:51,449 [MainThread] INFO EvalMetric - ARand: 0.7060739580961154
2023-02-03 13:31:51,486 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 13:31:59,106 [MainThread] INFO EvalMetric - ARand: 0.6601846897466341
2023-02-03 13:31:59,146 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 13:32:06,569 [MainThread] INFO EvalMetric - ARand: 0.6323125549181234
2023-02-03 13:32:06,607 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 13:32:13,894 [MainThread] INFO EvalMetric - ARand: 0.6225089497018108
2023-02-03 13:32:13,937 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 13:32:21,342 [MainThread] INFO EvalMetric - ARand: 0.7069864053060513
2023-02-03 13:32:21,379 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 13:32:28,769 [MainThread] INFO EvalMetric - ARand: 0.737834762007237
2023-02-03 13:32:28,811 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 13:32:36,289 [MainThread] INFO EvalMetric - ARand: 0.7120422758779488
2023-02-03 13:32:36,333 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 13:32:43,724 [MainThread] INFO EvalMetric - ARand: 0.6853314612978533
2023-02-03 13:32:43,764 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 13:32:51,193 [MainThread] INFO EvalMetric - ARand: 0.6812209198225432
2023-02-03 13:32:51,236 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 13:32:58,699 [MainThread] INFO EvalMetric - ARand: 0.6903287105600895
2023-02-03 13:32:58,743 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 13:33:06,167 [MainThread] INFO EvalMetric - ARand: 0.7001507204180553
2023-02-03 13:33:06,212 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 13:33:13,788 [MainThread] INFO EvalMetric - ARand: 0.6788385688710824
2023-02-03 13:33:13,836 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 13:33:21,390 [MainThread] INFO EvalMetric - ARand: 0.7513074992224125
2023-02-03 13:33:21,435 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 13:33:27,167 [MainThread] INFO EvalMetric - ARand: 0.7897030750264481
2023-02-03 13:33:27,538 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4799783175019012. Evaluation score: 0.6975183745393343
2023-02-03 13:33:27,562 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 13:33:35,041 [MainThread] INFO EvalMetric - ARand: 0.835650415123442
2023-02-03 13:33:35,060 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.19848330753544965. Evaluation score: 0.835650415123442
2023-02-03 13:33:35,060 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 13:33:36,158 [MainThread] INFO UNet3DTrainer - Training iteration [1001/60000]. Epoch [17/499]
2023-02-03 13:33:36,536 [MainThread] INFO UNet3DTrainer - Training iteration [1002/60000]. Epoch [17/499]
2023-02-03 13:33:37,056 [MainThread] INFO UNet3DTrainer - Training iteration [1003/60000]. Epoch [17/499]
2023-02-03 13:33:37,591 [MainThread] INFO UNet3DTrainer - Training iteration [1004/60000]. Epoch [17/499]
2023-02-03 13:33:38,122 [MainThread] INFO UNet3DTrainer - Training iteration [1005/60000]. Epoch [17/499]
2023-02-03 13:33:38,656 [MainThread] INFO UNet3DTrainer - Training iteration [1006/60000]. Epoch [17/499]
2023-02-03 13:33:39,182 [MainThread] INFO UNet3DTrainer - Training iteration [1007/60000]. Epoch [17/499]
2023-02-03 13:33:39,701 [MainThread] INFO UNet3DTrainer - Training iteration [1008/60000]. Epoch [17/499]
2023-02-03 13:33:47,553 [MainThread] INFO UNet3DTrainer - Training iteration [1009/60000]. Epoch [18/499]
2023-02-03 13:33:48,024 [MainThread] INFO UNet3DTrainer - Training iteration [1010/60000]. Epoch [18/499]
2023-02-03 13:34:10,936 [MainThread] INFO UNet3DTrainer - Training iteration [1011/60000]. Epoch [18/499]
2023-02-03 13:34:11,338 [MainThread] INFO UNet3DTrainer - Training iteration [1012/60000]. Epoch [18/499]
2023-02-03 13:34:11,891 [MainThread] INFO UNet3DTrainer - Training iteration [1013/60000]. Epoch [18/499]
2023-02-03 13:34:12,447 [MainThread] INFO UNet3DTrainer - Training iteration [1014/60000]. Epoch [18/499]
2023-02-03 13:34:13,008 [MainThread] INFO UNet3DTrainer - Training iteration [1015/60000]. Epoch [18/499]
2023-02-03 13:34:13,612 [MainThread] INFO UNet3DTrainer - Training iteration [1016/60000]. Epoch [18/499]
2023-02-03 13:34:15,920 [MainThread] INFO UNet3DTrainer - Training iteration [1017/60000]. Epoch [18/499]
2023-02-03 13:34:16,318 [MainThread] INFO UNet3DTrainer - Training iteration [1018/60000]. Epoch [18/499]
2023-02-03 13:34:35,837 [MainThread] INFO UNet3DTrainer - Training iteration [1019/60000]. Epoch [18/499]
2023-02-03 13:34:36,227 [MainThread] INFO UNet3DTrainer - Training iteration [1020/60000]. Epoch [18/499]
2023-02-03 13:34:36,778 [MainThread] INFO UNet3DTrainer - Training iteration [1021/60000]. Epoch [18/499]
2023-02-03 13:34:37,330 [MainThread] INFO UNet3DTrainer - Training iteration [1022/60000]. Epoch [18/499]
2023-02-03 13:35:04,147 [MainThread] INFO UNet3DTrainer - Training iteration [1023/60000]. Epoch [18/499]
2023-02-03 13:35:04,534 [MainThread] INFO UNet3DTrainer - Training iteration [1024/60000]. Epoch [18/499]
2023-02-03 13:35:05,085 [MainThread] INFO UNet3DTrainer - Training iteration [1025/60000]. Epoch [18/499]
2023-02-03 13:35:05,626 [MainThread] INFO UNet3DTrainer - Training iteration [1026/60000]. Epoch [18/499]
2023-02-03 13:35:06,177 [MainThread] INFO UNet3DTrainer - Training iteration [1027/60000]. Epoch [18/499]
2023-02-03 13:35:06,738 [MainThread] INFO UNet3DTrainer - Training iteration [1028/60000]. Epoch [18/499]
2023-02-03 13:35:27,370 [MainThread] INFO UNet3DTrainer - Training iteration [1029/60000]. Epoch [18/499]
2023-02-03 13:35:27,759 [MainThread] INFO UNet3DTrainer - Training iteration [1030/60000]. Epoch [18/499]
2023-02-03 13:35:28,302 [MainThread] INFO UNet3DTrainer - Training iteration [1031/60000]. Epoch [18/499]
2023-02-03 13:35:28,854 [MainThread] INFO UNet3DTrainer - Training iteration [1032/60000]. Epoch [18/499]
2023-02-03 13:35:29,393 [MainThread] INFO UNet3DTrainer - Training iteration [1033/60000]. Epoch [18/499]
2023-02-03 13:35:29,955 [MainThread] INFO UNet3DTrainer - Training iteration [1034/60000]. Epoch [18/499]
2023-02-03 13:35:50,076 [MainThread] INFO UNet3DTrainer - Training iteration [1035/60000]. Epoch [18/499]
2023-02-03 13:35:50,460 [MainThread] INFO UNet3DTrainer - Training iteration [1036/60000]. Epoch [18/499]
2023-02-03 13:35:51,011 [MainThread] INFO UNet3DTrainer - Training iteration [1037/60000]. Epoch [18/499]
2023-02-03 13:35:51,547 [MainThread] INFO UNet3DTrainer - Training iteration [1038/60000]. Epoch [18/499]
2023-02-03 13:35:52,087 [MainThread] INFO UNet3DTrainer - Training iteration [1039/60000]. Epoch [18/499]
2023-02-03 13:35:52,665 [MainThread] INFO UNet3DTrainer - Training iteration [1040/60000]. Epoch [18/499]
2023-02-03 13:35:54,871 [MainThread] INFO UNet3DTrainer - Training iteration [1041/60000]. Epoch [18/499]
2023-02-03 13:35:55,344 [MainThread] INFO UNet3DTrainer - Training iteration [1042/60000]. Epoch [18/499]
2023-02-03 13:36:12,781 [MainThread] INFO UNet3DTrainer - Training iteration [1043/60000]. Epoch [18/499]
2023-02-03 13:36:13,174 [MainThread] INFO UNet3DTrainer - Training iteration [1044/60000]. Epoch [18/499]
2023-02-03 13:36:13,736 [MainThread] INFO UNet3DTrainer - Training iteration [1045/60000]. Epoch [18/499]
2023-02-03 13:36:14,287 [MainThread] INFO UNet3DTrainer - Training iteration [1046/60000]. Epoch [18/499]
2023-02-03 13:36:14,880 [MainThread] INFO UNet3DTrainer - Training iteration [1047/60000]. Epoch [18/499]
2023-02-03 13:36:15,515 [MainThread] INFO UNet3DTrainer - Training iteration [1048/60000]. Epoch [18/499]
2023-02-03 13:36:17,759 [MainThread] INFO UNet3DTrainer - Training iteration [1049/60000]. Epoch [18/499]
2023-02-03 13:36:18,118 [MainThread] INFO UNet3DTrainer - Training iteration [1050/60000]. Epoch [18/499]
2023-02-03 13:36:19,456 [MainThread] INFO UNet3DTrainer - Training iteration [1051/60000]. Epoch [18/499]
2023-02-03 13:36:19,870 [MainThread] INFO UNet3DTrainer - Training iteration [1052/60000]. Epoch [18/499]
2023-02-03 13:36:27,691 [MainThread] INFO UNet3DTrainer - Training iteration [1053/60000]. Epoch [18/499]
2023-02-03 13:36:48,462 [MainThread] INFO UNet3DTrainer - Training iteration [1054/60000]. Epoch [18/499]
2023-02-03 13:36:48,878 [MainThread] INFO UNet3DTrainer - Training iteration [1055/60000]. Epoch [18/499]
2023-02-03 13:37:04,325 [MainThread] INFO UNet3DTrainer - Training iteration [1056/60000]. Epoch [18/499]
2023-02-03 13:37:04,712 [MainThread] INFO UNet3DTrainer - Training iteration [1057/60000]. Epoch [18/499]
2023-02-03 13:37:05,255 [MainThread] INFO UNet3DTrainer - Training iteration [1058/60000]. Epoch [18/499]
2023-02-03 13:37:32,648 [MainThread] INFO UNet3DTrainer - Training iteration [1059/60000]. Epoch [18/499]
2023-02-03 13:37:33,023 [MainThread] INFO UNet3DTrainer - Training iteration [1060/60000]. Epoch [18/499]
2023-02-03 13:37:33,540 [MainThread] INFO UNet3DTrainer - Training iteration [1061/60000]. Epoch [18/499]
2023-02-03 13:37:34,084 [MainThread] INFO UNet3DTrainer - Training iteration [1062/60000]. Epoch [18/499]
2023-02-03 13:37:34,614 [MainThread] INFO UNet3DTrainer - Training iteration [1063/60000]. Epoch [18/499]
2023-02-03 13:37:35,137 [MainThread] INFO UNet3DTrainer - Training iteration [1064/60000]. Epoch [18/499]
2023-02-03 13:38:05,108 [MainThread] INFO UNet3DTrainer - Training iteration [1065/60000]. Epoch [19/499]
2023-02-03 13:38:05,508 [MainThread] INFO UNet3DTrainer - Training iteration [1066/60000]. Epoch [19/499]
2023-02-03 13:38:06,060 [MainThread] INFO UNet3DTrainer - Training iteration [1067/60000]. Epoch [19/499]
2023-02-03 13:38:24,192 [MainThread] INFO UNet3DTrainer - Training iteration [1068/60000]. Epoch [19/499]
2023-02-03 13:38:24,580 [MainThread] INFO UNet3DTrainer - Training iteration [1069/60000]. Epoch [19/499]
2023-02-03 13:38:25,123 [MainThread] INFO UNet3DTrainer - Training iteration [1070/60000]. Epoch [19/499]
2023-02-03 13:38:25,677 [MainThread] INFO UNet3DTrainer - Training iteration [1071/60000]. Epoch [19/499]
2023-02-03 13:38:26,229 [MainThread] INFO UNet3DTrainer - Training iteration [1072/60000]. Epoch [19/499]
2023-02-03 13:38:26,778 [MainThread] INFO UNet3DTrainer - Training iteration [1073/60000]. Epoch [19/499]
2023-02-03 13:38:50,945 [MainThread] INFO UNet3DTrainer - Training iteration [1074/60000]. Epoch [19/499]
2023-02-03 13:38:51,343 [MainThread] INFO UNet3DTrainer - Training iteration [1075/60000]. Epoch [19/499]
2023-02-03 13:38:51,889 [MainThread] INFO UNet3DTrainer - Training iteration [1076/60000]. Epoch [19/499]
2023-02-03 13:38:52,449 [MainThread] INFO UNet3DTrainer - Training iteration [1077/60000]. Epoch [19/499]
2023-02-03 13:38:53,003 [MainThread] INFO UNet3DTrainer - Training iteration [1078/60000]. Epoch [19/499]
2023-02-03 13:38:53,554 [MainThread] INFO UNet3DTrainer - Training iteration [1079/60000]. Epoch [19/499]
2023-02-03 13:38:55,278 [MainThread] INFO UNet3DTrainer - Training iteration [1080/60000]. Epoch [19/499]
2023-02-03 13:38:55,662 [MainThread] INFO UNet3DTrainer - Training iteration [1081/60000]. Epoch [19/499]
2023-02-03 13:39:18,411 [MainThread] INFO UNet3DTrainer - Training iteration [1082/60000]. Epoch [19/499]
2023-02-03 13:39:18,813 [MainThread] INFO UNet3DTrainer - Training iteration [1083/60000]. Epoch [19/499]
2023-02-03 13:39:19,386 [MainThread] INFO UNet3DTrainer - Training iteration [1084/60000]. Epoch [19/499]
2023-02-03 13:39:20,070 [MainThread] INFO UNet3DTrainer - Training iteration [1085/60000]. Epoch [19/499]
2023-02-03 13:39:23,248 [MainThread] INFO UNet3DTrainer - Training iteration [1086/60000]. Epoch [19/499]
2023-02-03 13:39:23,720 [MainThread] INFO UNet3DTrainer - Training iteration [1087/60000]. Epoch [19/499]
2023-02-03 13:39:24,290 [MainThread] INFO UNet3DTrainer - Training iteration [1088/60000]. Epoch [19/499]
2023-02-03 13:39:24,859 [MainThread] INFO UNet3DTrainer - Training iteration [1089/60000]. Epoch [19/499]
2023-02-03 13:39:25,402 [MainThread] INFO UNet3DTrainer - Training iteration [1090/60000]. Epoch [19/499]
2023-02-03 13:39:25,964 [MainThread] INFO UNet3DTrainer - Training iteration [1091/60000]. Epoch [19/499]
2023-02-03 13:39:49,148 [MainThread] INFO UNet3DTrainer - Training iteration [1092/60000]. Epoch [19/499]
2023-02-03 13:39:49,536 [MainThread] INFO UNet3DTrainer - Training iteration [1093/60000]. Epoch [19/499]
2023-02-03 13:39:50,108 [MainThread] INFO UNet3DTrainer - Training iteration [1094/60000]. Epoch [19/499]
2023-02-03 13:39:50,648 [MainThread] INFO UNet3DTrainer - Training iteration [1095/60000]. Epoch [19/499]
2023-02-03 13:39:51,214 [MainThread] INFO UNet3DTrainer - Training iteration [1096/60000]. Epoch [19/499]
2023-02-03 13:39:51,830 [MainThread] INFO UNet3DTrainer - Training iteration [1097/60000]. Epoch [19/499]
2023-02-03 13:39:54,384 [MainThread] INFO UNet3DTrainer - Training iteration [1098/60000]. Epoch [19/499]
2023-02-03 13:40:12,227 [MainThread] INFO UNet3DTrainer - Training iteration [1099/60000]. Epoch [19/499]
2023-02-03 13:40:12,616 [MainThread] INFO UNet3DTrainer - Training iteration [1100/60000]. Epoch [19/499]
2023-02-03 13:40:13,150 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 13:40:19,306 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 13:40:30,795 [MainThread] INFO EvalMetric - ARand: 0.6953791504002318
2023-02-03 13:40:30,838 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 13:40:38,300 [MainThread] INFO EvalMetric - ARand: 0.7204774324390439
2023-02-03 13:40:38,355 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 13:40:46,008 [MainThread] INFO EvalMetric - ARand: 0.7342088997254446
2023-02-03 13:40:46,048 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 13:40:53,471 [MainThread] INFO EvalMetric - ARand: 0.6888081386026264
2023-02-03 13:40:53,510 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 13:41:00,996 [MainThread] INFO EvalMetric - ARand: 0.707357588850529
2023-02-03 13:41:01,035 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 13:41:08,376 [MainThread] INFO EvalMetric - ARand: 0.7444758756059463
2023-02-03 13:41:08,423 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 13:41:15,896 [MainThread] INFO EvalMetric - ARand: 0.6821823708914144
2023-02-03 13:41:15,938 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 13:41:23,395 [MainThread] INFO EvalMetric - ARand: 0.7519165516588843
2023-02-03 13:41:23,440 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 13:41:30,827 [MainThread] INFO EvalMetric - ARand: 0.7083518467590891
2023-02-03 13:41:30,868 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 13:41:38,314 [MainThread] INFO EvalMetric - ARand: 0.7019828399681491
2023-02-03 13:41:38,353 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 13:41:45,877 [MainThread] INFO EvalMetric - ARand: 0.6214574796580092
2023-02-03 13:41:45,915 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 13:41:53,357 [MainThread] INFO EvalMetric - ARand: 0.6366994362699705
2023-02-03 13:41:53,394 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 13:42:00,960 [MainThread] INFO EvalMetric - ARand: 0.7066105772428397
2023-02-03 13:42:01,005 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 13:42:08,487 [MainThread] INFO EvalMetric - ARand: 0.7722764046036342
2023-02-03 13:42:08,534 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 13:42:15,998 [MainThread] INFO EvalMetric - ARand: 0.6991120273852239
2023-02-03 13:42:16,037 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 13:42:23,562 [MainThread] INFO EvalMetric - ARand: 0.7289804143746621
2023-02-03 13:42:23,609 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 13:42:31,191 [MainThread] INFO EvalMetric - ARand: 0.6985595352278662
2023-02-03 13:42:31,235 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 13:42:38,721 [MainThread] INFO EvalMetric - ARand: 0.701239514505928
2023-02-03 13:42:38,764 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 13:42:46,169 [MainThread] INFO EvalMetric - ARand: 0.6816798328274806
2023-02-03 13:42:46,210 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 13:42:53,777 [MainThread] INFO EvalMetric - ARand: 0.6779338165796782
2023-02-03 13:42:53,822 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 13:43:01,292 [MainThread] INFO EvalMetric - ARand: 0.7574101818872376
2023-02-03 13:43:01,337 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 13:43:07,054 [MainThread] INFO EvalMetric - ARand: 0.7594882012006688
2023-02-03 13:43:07,448 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4803908830401542. Evaluation score: 0.7074352214420411
2023-02-03 13:43:07,473 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 13:43:14,872 [MainThread] INFO EvalMetric - ARand: 0.7922389258940691
2023-02-03 13:43:14,892 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.1937785078254011. Evaluation score: 0.7922389258940691
2023-02-03 13:43:14,893 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 13:43:15,930 [MainThread] INFO UNet3DTrainer - Training iteration [1101/60000]. Epoch [19/499]
2023-02-03 13:43:16,309 [MainThread] INFO UNet3DTrainer - Training iteration [1102/60000]. Epoch [19/499]
2023-02-03 13:43:16,834 [MainThread] INFO UNet3DTrainer - Training iteration [1103/60000]. Epoch [19/499]
2023-02-03 13:43:17,373 [MainThread] INFO UNet3DTrainer - Training iteration [1104/60000]. Epoch [19/499]
2023-02-03 13:43:17,933 [MainThread] INFO UNet3DTrainer - Training iteration [1105/60000]. Epoch [19/499]
2023-02-03 13:43:18,552 [MainThread] INFO UNet3DTrainer - Training iteration [1106/60000]. Epoch [19/499]
2023-02-03 13:43:19,082 [MainThread] INFO UNet3DTrainer - Training iteration [1107/60000]. Epoch [19/499]
2023-02-03 13:43:19,634 [MainThread] INFO UNet3DTrainer - Training iteration [1108/60000]. Epoch [19/499]
2023-02-03 13:43:20,208 [MainThread] INFO UNet3DTrainer - Training iteration [1109/60000]. Epoch [19/499]
2023-02-03 13:43:20,759 [MainThread] INFO UNet3DTrainer - Training iteration [1110/60000]. Epoch [19/499]
2023-02-03 13:43:21,333 [MainThread] INFO UNet3DTrainer - Training iteration [1111/60000]. Epoch [19/499]
2023-02-03 13:43:21,932 [MainThread] INFO UNet3DTrainer - Training iteration [1112/60000]. Epoch [19/499]
2023-02-03 13:43:39,262 [MainThread] INFO UNet3DTrainer - Training iteration [1113/60000]. Epoch [19/499]
2023-02-03 13:43:39,654 [MainThread] INFO UNet3DTrainer - Training iteration [1114/60000]. Epoch [19/499]
2023-02-03 13:43:40,727 [MainThread] INFO UNet3DTrainer - Training iteration [1115/60000]. Epoch [19/499]
2023-02-03 13:43:41,063 [MainThread] INFO UNet3DTrainer - Training iteration [1116/60000]. Epoch [19/499]
2023-02-03 13:43:41,598 [MainThread] INFO UNet3DTrainer - Training iteration [1117/60000]. Epoch [19/499]
2023-02-03 13:43:42,122 [MainThread] INFO UNet3DTrainer - Training iteration [1118/60000]. Epoch [19/499]
2023-02-03 13:44:18,670 [MainThread] INFO UNet3DTrainer - Training iteration [1119/60000]. Epoch [19/499]
2023-02-03 13:44:19,046 [MainThread] INFO UNet3DTrainer - Training iteration [1120/60000]. Epoch [19/499]
2023-02-03 13:44:27,190 [MainThread] INFO UNet3DTrainer - Training iteration [1121/60000]. Epoch [20/499]
2023-02-03 13:44:27,740 [MainThread] INFO UNet3DTrainer - Training iteration [1122/60000]. Epoch [20/499]
2023-02-03 13:44:52,270 [MainThread] INFO UNet3DTrainer - Training iteration [1123/60000]. Epoch [20/499]
2023-02-03 13:44:52,901 [MainThread] INFO UNet3DTrainer - Training iteration [1124/60000]. Epoch [20/499]
2023-02-03 13:44:53,374 [MainThread] INFO UNet3DTrainer - Training iteration [1125/60000]. Epoch [20/499]
2023-02-03 13:44:53,953 [MainThread] INFO UNet3DTrainer - Training iteration [1126/60000]. Epoch [20/499]
2023-02-03 13:44:54,564 [MainThread] INFO UNet3DTrainer - Training iteration [1127/60000]. Epoch [20/499]
2023-02-03 13:44:55,088 [MainThread] INFO UNet3DTrainer - Training iteration [1128/60000]. Epoch [20/499]
2023-02-03 13:44:57,209 [MainThread] INFO UNet3DTrainer - Training iteration [1129/60000]. Epoch [20/499]
2023-02-03 13:44:58,331 [MainThread] INFO UNet3DTrainer - Training iteration [1130/60000]. Epoch [20/499]
2023-02-03 13:45:20,008 [MainThread] INFO UNet3DTrainer - Training iteration [1131/60000]. Epoch [20/499]
2023-02-03 13:45:20,405 [MainThread] INFO UNet3DTrainer - Training iteration [1132/60000]. Epoch [20/499]
2023-02-03 13:45:20,969 [MainThread] INFO UNet3DTrainer - Training iteration [1133/60000]. Epoch [20/499]
2023-02-03 13:45:21,522 [MainThread] INFO UNet3DTrainer - Training iteration [1134/60000]. Epoch [20/499]
2023-02-03 13:45:22,084 [MainThread] INFO UNet3DTrainer - Training iteration [1135/60000]. Epoch [20/499]
2023-02-03 13:45:22,626 [MainThread] INFO UNet3DTrainer - Training iteration [1136/60000]. Epoch [20/499]
2023-02-03 13:45:24,128 [MainThread] INFO UNet3DTrainer - Training iteration [1137/60000]. Epoch [20/499]
2023-02-03 13:45:45,085 [MainThread] INFO UNet3DTrainer - Training iteration [1138/60000]. Epoch [20/499]
2023-02-03 13:45:45,457 [MainThread] INFO UNet3DTrainer - Training iteration [1139/60000]. Epoch [20/499]
2023-02-03 13:45:45,985 [MainThread] INFO UNet3DTrainer - Training iteration [1140/60000]. Epoch [20/499]
2023-02-03 13:45:46,535 [MainThread] INFO UNet3DTrainer - Training iteration [1141/60000]. Epoch [20/499]
2023-02-03 13:45:47,082 [MainThread] INFO UNet3DTrainer - Training iteration [1142/60000]. Epoch [20/499]
2023-02-03 13:45:47,631 [MainThread] INFO UNet3DTrainer - Training iteration [1143/60000]. Epoch [20/499]
2023-02-03 13:45:49,200 [MainThread] INFO UNet3DTrainer - Training iteration [1144/60000]. Epoch [20/499]
2023-02-03 13:45:49,656 [MainThread] INFO UNet3DTrainer - Training iteration [1145/60000]. Epoch [20/499]
2023-02-03 13:45:50,285 [MainThread] INFO UNet3DTrainer - Training iteration [1146/60000]. Epoch [20/499]
2023-02-03 13:45:50,924 [MainThread] INFO UNet3DTrainer - Training iteration [1147/60000]. Epoch [20/499]
2023-02-03 13:45:51,457 [MainThread] INFO UNet3DTrainer - Training iteration [1148/60000]. Epoch [20/499]
2023-02-03 13:45:52,026 [MainThread] INFO UNet3DTrainer - Training iteration [1149/60000]. Epoch [20/499]
2023-02-03 13:46:19,460 [MainThread] INFO UNet3DTrainer - Training iteration [1150/60000]. Epoch [20/499]
2023-02-03 13:46:20,010 [MainThread] INFO UNet3DTrainer - Training iteration [1151/60000]. Epoch [20/499]
2023-02-03 13:46:36,632 [MainThread] INFO UNet3DTrainer - Training iteration [1152/60000]. Epoch [20/499]
2023-02-03 13:46:37,033 [MainThread] INFO UNet3DTrainer - Training iteration [1153/60000]. Epoch [20/499]
2023-02-03 13:46:37,556 [MainThread] INFO UNet3DTrainer - Training iteration [1154/60000]. Epoch [20/499]
2023-02-03 13:46:38,134 [MainThread] INFO UNet3DTrainer - Training iteration [1155/60000]. Epoch [20/499]
2023-02-03 13:46:43,479 [MainThread] INFO UNet3DTrainer - Training iteration [1156/60000]. Epoch [20/499]
2023-02-03 13:46:43,955 [MainThread] INFO UNet3DTrainer - Training iteration [1157/60000]. Epoch [20/499]
2023-02-03 13:46:44,630 [MainThread] INFO UNet3DTrainer - Training iteration [1158/60000]. Epoch [20/499]
2023-02-03 13:46:45,320 [MainThread] INFO UNet3DTrainer - Training iteration [1159/60000]. Epoch [20/499]
2023-02-03 13:46:45,922 [MainThread] INFO UNet3DTrainer - Training iteration [1160/60000]. Epoch [20/499]
2023-02-03 13:46:46,539 [MainThread] INFO UNet3DTrainer - Training iteration [1161/60000]. Epoch [20/499]
2023-02-03 13:46:48,830 [MainThread] INFO UNet3DTrainer - Training iteration [1162/60000]. Epoch [20/499]
2023-02-03 13:47:12,075 [MainThread] INFO UNet3DTrainer - Training iteration [1163/60000]. Epoch [20/499]
2023-02-03 13:47:32,415 [MainThread] INFO UNet3DTrainer - Training iteration [1164/60000]. Epoch [20/499]
2023-02-03 13:47:32,805 [MainThread] INFO UNet3DTrainer - Training iteration [1165/60000]. Epoch [20/499]
2023-02-03 13:47:33,342 [MainThread] INFO UNet3DTrainer - Training iteration [1166/60000]. Epoch [20/499]
2023-02-03 13:47:33,872 [MainThread] INFO UNet3DTrainer - Training iteration [1167/60000]. Epoch [20/499]
2023-02-03 13:47:34,401 [MainThread] INFO UNet3DTrainer - Training iteration [1168/60000]. Epoch [20/499]
2023-02-03 13:47:34,926 [MainThread] INFO UNet3DTrainer - Training iteration [1169/60000]. Epoch [20/499]
2023-02-03 13:47:54,143 [MainThread] INFO UNet3DTrainer - Training iteration [1170/60000]. Epoch [20/499]
2023-02-03 13:47:54,534 [MainThread] INFO UNet3DTrainer - Training iteration [1171/60000]. Epoch [20/499]
2023-02-03 13:47:55,052 [MainThread] INFO UNet3DTrainer - Training iteration [1172/60000]. Epoch [20/499]
2023-02-03 13:47:55,589 [MainThread] INFO UNet3DTrainer - Training iteration [1173/60000]. Epoch [20/499]
2023-02-03 13:47:56,113 [MainThread] INFO UNet3DTrainer - Training iteration [1174/60000]. Epoch [20/499]
2023-02-03 13:47:56,639 [MainThread] INFO UNet3DTrainer - Training iteration [1175/60000]. Epoch [20/499]
2023-02-03 13:48:14,479 [MainThread] INFO UNet3DTrainer - Training iteration [1176/60000]. Epoch [20/499]
2023-02-03 13:48:43,460 [MainThread] INFO UNet3DTrainer - Training iteration [1177/60000]. Epoch [21/499]
2023-02-03 13:48:43,842 [MainThread] INFO UNet3DTrainer - Training iteration [1178/60000]. Epoch [21/499]
2023-02-03 13:48:44,397 [MainThread] INFO UNet3DTrainer - Training iteration [1179/60000]. Epoch [21/499]
2023-02-03 13:48:44,963 [MainThread] INFO UNet3DTrainer - Training iteration [1180/60000]. Epoch [21/499]
2023-02-03 13:48:45,508 [MainThread] INFO UNet3DTrainer - Training iteration [1181/60000]. Epoch [21/499]
2023-02-03 13:48:46,090 [MainThread] INFO UNet3DTrainer - Training iteration [1182/60000]. Epoch [21/499]
2023-02-03 13:48:48,162 [MainThread] INFO UNet3DTrainer - Training iteration [1183/60000]. Epoch [21/499]
2023-02-03 13:48:48,610 [MainThread] INFO UNet3DTrainer - Training iteration [1184/60000]. Epoch [21/499]
2023-02-03 13:48:49,940 [MainThread] INFO UNet3DTrainer - Training iteration [1185/60000]. Epoch [21/499]
2023-02-03 13:49:10,779 [MainThread] INFO UNet3DTrainer - Training iteration [1186/60000]. Epoch [21/499]
2023-02-03 13:49:11,175 [MainThread] INFO UNet3DTrainer - Training iteration [1187/60000]. Epoch [21/499]
2023-02-03 13:49:11,715 [MainThread] INFO UNet3DTrainer - Training iteration [1188/60000]. Epoch [21/499]
2023-02-03 13:49:12,261 [MainThread] INFO UNet3DTrainer - Training iteration [1189/60000]. Epoch [21/499]
2023-02-03 13:49:12,820 [MainThread] INFO UNet3DTrainer - Training iteration [1190/60000]. Epoch [21/499]
2023-02-03 13:49:15,991 [MainThread] INFO UNet3DTrainer - Training iteration [1191/60000]. Epoch [21/499]
2023-02-03 13:49:16,508 [MainThread] INFO UNet3DTrainer - Training iteration [1192/60000]. Epoch [21/499]
2023-02-03 13:49:35,944 [MainThread] INFO UNet3DTrainer - Training iteration [1193/60000]. Epoch [21/499]
2023-02-03 13:49:55,742 [MainThread] INFO UNet3DTrainer - Training iteration [1194/60000]. Epoch [21/499]
2023-02-03 13:49:56,129 [MainThread] INFO UNet3DTrainer - Training iteration [1195/60000]. Epoch [21/499]
2023-02-03 13:49:56,673 [MainThread] INFO UNet3DTrainer - Training iteration [1196/60000]. Epoch [21/499]
2023-02-03 13:49:57,240 [MainThread] INFO UNet3DTrainer - Training iteration [1197/60000]. Epoch [21/499]
2023-02-03 13:49:57,813 [MainThread] INFO UNet3DTrainer - Training iteration [1198/60000]. Epoch [21/499]
2023-02-03 13:49:58,592 [MainThread] INFO UNet3DTrainer - Training iteration [1199/60000]. Epoch [21/499]
2023-02-03 13:50:00,321 [MainThread] INFO UNet3DTrainer - Training iteration [1200/60000]. Epoch [21/499]
2023-02-03 13:50:00,652 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 13:50:06,861 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 13:50:18,742 [MainThread] INFO EvalMetric - ARand: 0.6497261695983486
2023-02-03 13:50:18,784 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 13:50:27,681 [MainThread] INFO EvalMetric - ARand: 0.6539730317646428
2023-02-03 13:50:27,722 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 13:50:36,155 [MainThread] INFO EvalMetric - ARand: 0.7115275530274892
2023-02-03 13:50:36,199 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 13:50:44,479 [MainThread] INFO EvalMetric - ARand: 0.608273723687006
2023-02-03 13:50:44,528 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 13:50:52,562 [MainThread] INFO EvalMetric - ARand: 0.6340375114490169
2023-02-03 13:50:52,608 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 13:51:00,401 [MainThread] INFO EvalMetric - ARand: 0.6925049413311863
2023-02-03 13:51:00,453 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 13:51:07,924 [MainThread] INFO EvalMetric - ARand: 0.5611432841289902
2023-02-03 13:51:07,965 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 13:51:15,346 [MainThread] INFO EvalMetric - ARand: 0.6273706547424539
2023-02-03 13:51:15,389 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 13:51:22,762 [MainThread] INFO EvalMetric - ARand: 0.6545673792826308
2023-02-03 13:51:22,803 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 13:51:30,277 [MainThread] INFO EvalMetric - ARand: 0.6419327928747969
2023-02-03 13:51:30,315 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 13:51:37,726 [MainThread] INFO EvalMetric - ARand: 0.5707248207897263
2023-02-03 13:51:37,767 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 13:51:45,234 [MainThread] INFO EvalMetric - ARand: 0.5323124230168639
2023-02-03 13:51:45,276 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 13:51:52,714 [MainThread] INFO EvalMetric - ARand: 0.6461958841782163
2023-02-03 13:51:52,751 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 13:52:00,247 [MainThread] INFO EvalMetric - ARand: 0.6445382947174141
2023-02-03 13:52:00,293 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 13:52:07,758 [MainThread] INFO EvalMetric - ARand: 0.5643263174808252
2023-02-03 13:52:07,804 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 13:52:15,286 [MainThread] INFO EvalMetric - ARand: 0.6217328179372876
2023-02-03 13:52:15,325 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 13:52:22,806 [MainThread] INFO EvalMetric - ARand: 0.60956201287935
2023-02-03 13:52:22,849 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 13:52:30,343 [MainThread] INFO EvalMetric - ARand: 0.620635700466994
2023-02-03 13:52:30,391 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 13:52:37,902 [MainThread] INFO EvalMetric - ARand: 0.645017679555943
2023-02-03 13:52:37,947 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 13:52:45,494 [MainThread] INFO EvalMetric - ARand: 0.6244555247998831
2023-02-03 13:52:45,538 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 13:52:53,004 [MainThread] INFO EvalMetric - ARand: 0.7310096352431301
2023-02-03 13:52:53,048 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 13:52:58,729 [MainThread] INFO EvalMetric - ARand: 0.7166496327034132
2023-02-03 13:52:59,116 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4782643619625049. Evaluation score: 0.6337036955163105
2023-02-03 13:52:59,139 [MainThread] INFO UNet3DTrainer - Saving new best evaluation metric: 0.6337036955163105
2023-02-03 13:52:59,141 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 13:53:06,811 [MainThread] INFO EvalMetric - ARand: 0.733574944298246
2023-02-03 13:53:06,832 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.18602215747038522. Evaluation score: 0.733574944298246
2023-02-03 13:53:06,833 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 13:53:07,887 [MainThread] INFO UNet3DTrainer - Training iteration [1201/60000]. Epoch [21/499]
2023-02-03 13:53:08,249 [MainThread] INFO UNet3DTrainer - Training iteration [1202/60000]. Epoch [21/499]
2023-02-03 13:53:08,784 [MainThread] INFO UNet3DTrainer - Training iteration [1203/60000]. Epoch [21/499]
2023-02-03 13:53:09,322 [MainThread] INFO UNet3DTrainer - Training iteration [1204/60000]. Epoch [21/499]
2023-02-03 13:53:09,876 [MainThread] INFO UNet3DTrainer - Training iteration [1205/60000]. Epoch [21/499]
2023-02-03 13:53:10,428 [MainThread] INFO UNet3DTrainer - Training iteration [1206/60000]. Epoch [21/499]
2023-02-03 13:53:11,054 [MainThread] INFO UNet3DTrainer - Training iteration [1207/60000]. Epoch [21/499]
2023-02-03 13:53:11,704 [MainThread] INFO UNet3DTrainer - Training iteration [1208/60000]. Epoch [21/499]
2023-02-03 13:53:12,264 [MainThread] INFO UNet3DTrainer - Training iteration [1209/60000]. Epoch [21/499]
2023-02-03 13:53:12,930 [MainThread] INFO UNet3DTrainer - Training iteration [1210/60000]. Epoch [21/499]
2023-02-03 13:53:13,512 [MainThread] INFO UNet3DTrainer - Training iteration [1211/60000]. Epoch [21/499]
2023-02-03 13:53:14,180 [MainThread] INFO UNet3DTrainer - Training iteration [1212/60000]. Epoch [21/499]
2023-02-03 13:53:31,898 [MainThread] INFO UNet3DTrainer - Training iteration [1213/60000]. Epoch [21/499]
2023-02-03 13:53:32,270 [MainThread] INFO UNet3DTrainer - Training iteration [1214/60000]. Epoch [21/499]
2023-02-03 13:53:33,755 [MainThread] INFO UNet3DTrainer - Training iteration [1215/60000]. Epoch [21/499]
2023-02-03 13:53:34,111 [MainThread] INFO UNet3DTrainer - Training iteration [1216/60000]. Epoch [21/499]
2023-02-03 13:53:35,777 [MainThread] INFO UNet3DTrainer - Training iteration [1217/60000]. Epoch [21/499]
2023-02-03 13:53:36,161 [MainThread] INFO UNet3DTrainer - Training iteration [1218/60000]. Epoch [21/499]
2023-02-03 13:53:36,718 [MainThread] INFO UNet3DTrainer - Training iteration [1219/60000]. Epoch [21/499]
2023-02-03 13:53:37,290 [MainThread] INFO UNet3DTrainer - Training iteration [1220/60000]. Epoch [21/499]
2023-02-03 13:54:22,724 [MainThread] INFO UNet3DTrainer - Training iteration [1221/60000]. Epoch [21/499]
2023-02-03 13:54:23,090 [MainThread] INFO UNet3DTrainer - Training iteration [1222/60000]. Epoch [21/499]
2023-02-03 13:54:23,620 [MainThread] INFO UNet3DTrainer - Training iteration [1223/60000]. Epoch [21/499]
2023-02-03 13:54:24,156 [MainThread] INFO UNet3DTrainer - Training iteration [1224/60000]. Epoch [21/499]
2023-02-03 13:54:24,685 [MainThread] INFO UNet3DTrainer - Training iteration [1225/60000]. Epoch [21/499]
2023-02-03 13:54:25,217 [MainThread] INFO UNet3DTrainer - Training iteration [1226/60000]. Epoch [21/499]
2023-02-03 13:54:25,750 [MainThread] INFO UNet3DTrainer - Training iteration [1227/60000]. Epoch [21/499]
2023-02-03 13:54:26,285 [MainThread] INFO UNet3DTrainer - Training iteration [1228/60000]. Epoch [21/499]
2023-02-03 13:54:26,812 [MainThread] INFO UNet3DTrainer - Training iteration [1229/60000]. Epoch [21/499]
2023-02-03 13:54:27,344 [MainThread] INFO UNet3DTrainer - Training iteration [1230/60000]. Epoch [21/499]
2023-02-03 13:54:27,876 [MainThread] INFO UNet3DTrainer - Training iteration [1231/60000]. Epoch [21/499]
2023-02-03 13:54:28,406 [MainThread] INFO UNet3DTrainer - Training iteration [1232/60000]. Epoch [21/499]
2023-02-03 13:54:36,355 [MainThread] INFO UNet3DTrainer - Training iteration [1233/60000]. Epoch [22/499]
2023-02-03 13:54:36,841 [MainThread] INFO UNet3DTrainer - Training iteration [1234/60000]. Epoch [22/499]
2023-02-03 13:55:01,777 [MainThread] INFO UNet3DTrainer - Training iteration [1235/60000]. Epoch [22/499]
2023-02-03 13:55:02,230 [MainThread] INFO UNet3DTrainer - Training iteration [1236/60000]. Epoch [22/499]
2023-02-03 13:55:22,070 [MainThread] INFO UNet3DTrainer - Training iteration [1237/60000]. Epoch [22/499]
2023-02-03 13:55:22,447 [MainThread] INFO UNet3DTrainer - Training iteration [1238/60000]. Epoch [22/499]
2023-02-03 13:55:23,003 [MainThread] INFO UNet3DTrainer - Training iteration [1239/60000]. Epoch [22/499]
2023-02-03 13:55:23,538 [MainThread] INFO UNet3DTrainer - Training iteration [1240/60000]. Epoch [22/499]
2023-02-03 13:55:26,435 [MainThread] INFO UNet3DTrainer - Training iteration [1241/60000]. Epoch [22/499]
2023-02-03 13:55:26,841 [MainThread] INFO UNet3DTrainer - Training iteration [1242/60000]. Epoch [22/499]
2023-02-03 13:55:27,402 [MainThread] INFO UNet3DTrainer - Training iteration [1243/60000]. Epoch [22/499]
2023-02-03 13:55:27,945 [MainThread] INFO UNet3DTrainer - Training iteration [1244/60000]. Epoch [22/499]
2023-02-03 13:55:28,620 [MainThread] INFO UNet3DTrainer - Training iteration [1245/60000]. Epoch [22/499]
2023-02-03 13:55:31,300 [MainThread] INFO UNet3DTrainer - Training iteration [1246/60000]. Epoch [22/499]
2023-02-03 13:55:52,716 [MainThread] INFO UNet3DTrainer - Training iteration [1247/60000]. Epoch [22/499]
2023-02-03 13:55:53,103 [MainThread] INFO UNet3DTrainer - Training iteration [1248/60000]. Epoch [22/499]
2023-02-03 13:55:53,652 [MainThread] INFO UNet3DTrainer - Training iteration [1249/60000]. Epoch [22/499]
2023-02-03 13:55:54,247 [MainThread] INFO UNet3DTrainer - Training iteration [1250/60000]. Epoch [22/499]
2023-02-03 13:55:54,813 [MainThread] INFO UNet3DTrainer - Training iteration [1251/60000]. Epoch [22/499]
2023-02-03 13:55:55,443 [MainThread] INFO UNet3DTrainer - Training iteration [1252/60000]. Epoch [22/499]
2023-02-03 13:55:58,691 [MainThread] INFO UNet3DTrainer - Training iteration [1253/60000]. Epoch [22/499]
2023-02-03 13:55:59,102 [MainThread] INFO UNet3DTrainer - Training iteration [1254/60000]. Epoch [22/499]
2023-02-03 13:55:59,702 [MainThread] INFO UNet3DTrainer - Training iteration [1255/60000]. Epoch [22/499]
2023-02-03 13:56:00,232 [MainThread] INFO UNet3DTrainer - Training iteration [1256/60000]. Epoch [22/499]
2023-02-03 13:56:18,734 [MainThread] INFO UNet3DTrainer - Training iteration [1257/60000]. Epoch [22/499]
2023-02-03 13:56:19,125 [MainThread] INFO UNet3DTrainer - Training iteration [1258/60000]. Epoch [22/499]
2023-02-03 13:56:27,389 [MainThread] INFO UNet3DTrainer - Training iteration [1259/60000]. Epoch [22/499]
2023-02-03 13:56:27,809 [MainThread] INFO UNet3DTrainer - Training iteration [1260/60000]. Epoch [22/499]
2023-02-03 13:56:47,368 [MainThread] INFO UNet3DTrainer - Training iteration [1261/60000]. Epoch [22/499]
2023-02-03 13:56:49,052 [MainThread] INFO UNet3DTrainer - Training iteration [1262/60000]. Epoch [22/499]
2023-02-03 13:56:49,414 [MainThread] INFO UNet3DTrainer - Training iteration [1263/60000]. Epoch [22/499]
2023-02-03 13:56:49,966 [MainThread] INFO UNet3DTrainer - Training iteration [1264/60000]. Epoch [22/499]
2023-02-03 13:56:55,842 [MainThread] INFO UNet3DTrainer - Training iteration [1265/60000]. Epoch [22/499]
2023-02-03 13:56:56,222 [MainThread] INFO UNet3DTrainer - Training iteration [1266/60000]. Epoch [22/499]
2023-02-03 13:56:56,774 [MainThread] INFO UNet3DTrainer - Training iteration [1267/60000]. Epoch [22/499]
2023-02-03 13:57:30,398 [MainThread] INFO UNet3DTrainer - Training iteration [1268/60000]. Epoch [22/499]
2023-02-03 13:57:30,757 [MainThread] INFO UNet3DTrainer - Training iteration [1269/60000]. Epoch [22/499]
2023-02-03 13:57:31,292 [MainThread] INFO UNet3DTrainer - Training iteration [1270/60000]. Epoch [22/499]
2023-02-03 13:57:31,845 [MainThread] INFO UNet3DTrainer - Training iteration [1271/60000]. Epoch [22/499]
2023-02-03 13:57:32,387 [MainThread] INFO UNet3DTrainer - Training iteration [1272/60000]. Epoch [22/499]
2023-02-03 13:57:32,945 [MainThread] INFO UNet3DTrainer - Training iteration [1273/60000]. Epoch [22/499]
2023-02-03 13:58:10,874 [MainThread] INFO UNet3DTrainer - Training iteration [1274/60000]. Epoch [22/499]
2023-02-03 13:58:11,251 [MainThread] INFO UNet3DTrainer - Training iteration [1275/60000]. Epoch [22/499]
2023-02-03 13:58:11,789 [MainThread] INFO UNet3DTrainer - Training iteration [1276/60000]. Epoch [22/499]
2023-02-03 13:58:12,331 [MainThread] INFO UNet3DTrainer - Training iteration [1277/60000]. Epoch [22/499]
2023-02-03 13:58:12,859 [MainThread] INFO UNet3DTrainer - Training iteration [1278/60000]. Epoch [22/499]
2023-02-03 13:58:13,418 [MainThread] INFO UNet3DTrainer - Training iteration [1279/60000]. Epoch [22/499]
2023-02-03 13:58:14,069 [MainThread] INFO UNet3DTrainer - Training iteration [1280/60000]. Epoch [22/499]
2023-02-03 13:58:14,500 [MainThread] INFO UNet3DTrainer - Training iteration [1281/60000]. Epoch [22/499]
2023-02-03 13:58:15,022 [MainThread] INFO UNet3DTrainer - Training iteration [1282/60000]. Epoch [22/499]
2023-02-03 13:58:15,553 [MainThread] INFO UNet3DTrainer - Training iteration [1283/60000]. Epoch [22/499]
2023-02-03 13:58:16,087 [MainThread] INFO UNet3DTrainer - Training iteration [1284/60000]. Epoch [22/499]
2023-02-03 13:58:16,615 [MainThread] INFO UNet3DTrainer - Training iteration [1285/60000]. Epoch [22/499]
2023-02-03 13:58:35,083 [MainThread] INFO UNet3DTrainer - Training iteration [1286/60000]. Epoch [22/499]
2023-02-03 13:58:35,461 [MainThread] INFO UNet3DTrainer - Training iteration [1287/60000]. Epoch [22/499]
2023-02-03 13:58:35,987 [MainThread] INFO UNet3DTrainer - Training iteration [1288/60000]. Epoch [22/499]
2023-02-03 13:59:02,496 [MainThread] INFO UNet3DTrainer - Training iteration [1289/60000]. Epoch [23/499]
2023-02-03 13:59:03,240 [MainThread] INFO UNet3DTrainer - Training iteration [1290/60000]. Epoch [23/499]
2023-02-03 13:59:03,588 [MainThread] INFO UNet3DTrainer - Training iteration [1291/60000]. Epoch [23/499]
2023-02-03 13:59:04,128 [MainThread] INFO UNet3DTrainer - Training iteration [1292/60000]. Epoch [23/499]
2023-02-03 13:59:04,710 [MainThread] INFO UNet3DTrainer - Training iteration [1293/60000]. Epoch [23/499]
2023-02-03 13:59:23,129 [MainThread] INFO UNet3DTrainer - Training iteration [1294/60000]. Epoch [23/499]
2023-02-03 13:59:23,518 [MainThread] INFO UNet3DTrainer - Training iteration [1295/60000]. Epoch [23/499]
2023-02-03 13:59:24,064 [MainThread] INFO UNet3DTrainer - Training iteration [1296/60000]. Epoch [23/499]
2023-02-03 13:59:24,606 [MainThread] INFO UNet3DTrainer - Training iteration [1297/60000]. Epoch [23/499]
2023-02-03 13:59:25,166 [MainThread] INFO UNet3DTrainer - Training iteration [1298/60000]. Epoch [23/499]
2023-02-03 13:59:25,725 [MainThread] INFO UNet3DTrainer - Training iteration [1299/60000]. Epoch [23/499]
2023-02-03 13:59:49,293 [MainThread] INFO UNet3DTrainer - Training iteration [1300/60000]. Epoch [23/499]
2023-02-03 13:59:49,649 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 13:59:55,049 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 14:00:05,322 [MainThread] INFO EvalMetric - ARand: 0.65208718867851
2023-02-03 14:00:05,366 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 14:00:12,804 [MainThread] INFO EvalMetric - ARand: 0.6914640242392454
2023-02-03 14:00:12,844 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 14:00:20,233 [MainThread] INFO EvalMetric - ARand: 0.7452973967606593
2023-02-03 14:00:20,271 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 14:00:27,625 [MainThread] INFO EvalMetric - ARand: 0.6909720311527903
2023-02-03 14:00:27,667 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 14:00:35,049 [MainThread] INFO EvalMetric - ARand: 0.699838015891304
2023-02-03 14:00:35,087 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 14:00:42,511 [MainThread] INFO EvalMetric - ARand: 0.7103949190566062
2023-02-03 14:00:42,552 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 14:00:50,026 [MainThread] INFO EvalMetric - ARand: 0.6810367176089589
2023-02-03 14:00:50,065 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 14:00:57,415 [MainThread] INFO EvalMetric - ARand: 0.740312603835274
2023-02-03 14:00:57,458 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 14:01:04,807 [MainThread] INFO EvalMetric - ARand: 0.6900970070112736
2023-02-03 14:01:04,847 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 14:01:12,285 [MainThread] INFO EvalMetric - ARand: 0.6996566449702235
2023-02-03 14:01:12,326 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 14:01:19,859 [MainThread] INFO EvalMetric - ARand: 0.6010025225940071
2023-02-03 14:01:19,895 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 14:01:27,331 [MainThread] INFO EvalMetric - ARand: 0.6499816643994759
2023-02-03 14:01:27,373 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 14:01:34,782 [MainThread] INFO EvalMetric - ARand: 0.6862889768047828
2023-02-03 14:01:34,819 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 14:01:42,199 [MainThread] INFO EvalMetric - ARand: 0.726754527170092
2023-02-03 14:01:42,239 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 14:01:49,642 [MainThread] INFO EvalMetric - ARand: 0.7077749925710208
2023-02-03 14:01:49,686 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 14:01:57,299 [MainThread] INFO EvalMetric - ARand: 0.7227503906687496
2023-02-03 14:01:57,340 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 14:02:04,739 [MainThread] INFO EvalMetric - ARand: 0.6916372217949002
2023-02-03 14:02:04,782 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 14:02:12,217 [MainThread] INFO EvalMetric - ARand: 0.6955830989622162
2023-02-03 14:02:12,263 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 14:02:19,784 [MainThread] INFO EvalMetric - ARand: 0.671838926858084
2023-02-03 14:02:19,830 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 14:02:27,314 [MainThread] INFO EvalMetric - ARand: 0.6821762024262801
2023-02-03 14:02:27,360 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 14:02:34,752 [MainThread] INFO EvalMetric - ARand: 0.7551289925092821
2023-02-03 14:02:34,797 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 14:02:40,365 [MainThread] INFO EvalMetric - ARand: 0.7904726971204031
2023-02-03 14:02:40,753 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4778989950815837. Evaluation score: 0.6981576362668523
2023-02-03 14:02:40,776 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 14:02:48,040 [MainThread] INFO EvalMetric - ARand: 0.7973471300424438
2023-02-03 14:02:48,060 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.16743157617747784. Evaluation score: 0.7973471300424438
2023-02-03 14:02:48,062 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 14:02:49,120 [MainThread] INFO UNet3DTrainer - Training iteration [1301/60000]. Epoch [23/499]
2023-02-03 14:02:49,494 [MainThread] INFO UNet3DTrainer - Training iteration [1302/60000]. Epoch [23/499]
2023-02-03 14:02:50,018 [MainThread] INFO UNet3DTrainer - Training iteration [1303/60000]. Epoch [23/499]
2023-02-03 14:02:50,562 [MainThread] INFO UNet3DTrainer - Training iteration [1304/60000]. Epoch [23/499]
2023-02-03 14:02:51,112 [MainThread] INFO UNet3DTrainer - Training iteration [1305/60000]. Epoch [23/499]
2023-02-03 14:02:51,670 [MainThread] INFO UNet3DTrainer - Training iteration [1306/60000]. Epoch [23/499]
2023-02-03 14:02:52,230 [MainThread] INFO UNet3DTrainer - Training iteration [1307/60000]. Epoch [23/499]
2023-02-03 14:02:52,830 [MainThread] INFO UNet3DTrainer - Training iteration [1308/60000]. Epoch [23/499]
2023-02-03 14:02:53,465 [MainThread] INFO UNet3DTrainer - Training iteration [1309/60000]. Epoch [23/499]
2023-02-03 14:02:54,051 [MainThread] INFO UNet3DTrainer - Training iteration [1310/60000]. Epoch [23/499]
2023-02-03 14:02:54,640 [MainThread] INFO UNet3DTrainer - Training iteration [1311/60000]. Epoch [23/499]
2023-02-03 14:02:55,230 [MainThread] INFO UNet3DTrainer - Training iteration [1312/60000]. Epoch [23/499]
2023-02-03 14:03:16,935 [MainThread] INFO UNet3DTrainer - Training iteration [1313/60000]. Epoch [23/499]
2023-02-03 14:03:17,333 [MainThread] INFO UNet3DTrainer - Training iteration [1314/60000]. Epoch [23/499]
2023-02-03 14:03:17,896 [MainThread] INFO UNet3DTrainer - Training iteration [1315/60000]. Epoch [23/499]
2023-02-03 14:03:18,570 [MainThread] INFO UNet3DTrainer - Training iteration [1316/60000]. Epoch [23/499]
2023-02-03 14:03:19,140 [MainThread] INFO UNet3DTrainer - Training iteration [1317/60000]. Epoch [23/499]
2023-02-03 14:03:20,725 [MainThread] INFO UNet3DTrainer - Training iteration [1318/60000]. Epoch [23/499]
2023-02-03 14:04:01,693 [MainThread] INFO UNet3DTrainer - Training iteration [1319/60000]. Epoch [23/499]
2023-02-03 14:04:02,071 [MainThread] INFO UNet3DTrainer - Training iteration [1320/60000]. Epoch [23/499]
2023-02-03 14:04:02,612 [MainThread] INFO UNet3DTrainer - Training iteration [1321/60000]. Epoch [23/499]
2023-02-03 14:04:03,161 [MainThread] INFO UNet3DTrainer - Training iteration [1322/60000]. Epoch [23/499]
2023-02-03 14:04:03,717 [MainThread] INFO UNet3DTrainer - Training iteration [1323/60000]. Epoch [23/499]
2023-02-03 14:04:04,337 [MainThread] INFO UNet3DTrainer - Training iteration [1324/60000]. Epoch [23/499]
2023-02-03 14:04:06,654 [MainThread] INFO UNet3DTrainer - Training iteration [1325/60000]. Epoch [23/499]
2023-02-03 14:04:07,076 [MainThread] INFO UNet3DTrainer - Training iteration [1326/60000]. Epoch [23/499]
2023-02-03 14:04:07,643 [MainThread] INFO UNet3DTrainer - Training iteration [1327/60000]. Epoch [23/499]
2023-02-03 14:04:08,273 [MainThread] INFO UNet3DTrainer - Training iteration [1328/60000]. Epoch [23/499]
2023-02-03 14:04:08,801 [MainThread] INFO UNet3DTrainer - Training iteration [1329/60000]. Epoch [23/499]
2023-02-03 14:04:29,737 [MainThread] INFO UNet3DTrainer - Training iteration [1330/60000]. Epoch [23/499]
2023-02-03 14:04:30,117 [MainThread] INFO UNet3DTrainer - Training iteration [1331/60000]. Epoch [23/499]
2023-02-03 14:04:30,654 [MainThread] INFO UNet3DTrainer - Training iteration [1332/60000]. Epoch [23/499]
2023-02-03 14:05:04,540 [MainThread] INFO UNet3DTrainer - Training iteration [1333/60000]. Epoch [23/499]
2023-02-03 14:05:04,908 [MainThread] INFO UNet3DTrainer - Training iteration [1334/60000]. Epoch [23/499]
2023-02-03 14:05:05,446 [MainThread] INFO UNet3DTrainer - Training iteration [1335/60000]. Epoch [23/499]
2023-02-03 14:05:27,912 [MainThread] INFO UNet3DTrainer - Training iteration [1336/60000]. Epoch [23/499]
2023-02-03 14:05:28,286 [MainThread] INFO UNet3DTrainer - Training iteration [1337/60000]. Epoch [23/499]
2023-02-03 14:05:28,808 [MainThread] INFO UNet3DTrainer - Training iteration [1338/60000]. Epoch [23/499]
2023-02-03 14:05:29,347 [MainThread] INFO UNet3DTrainer - Training iteration [1339/60000]. Epoch [23/499]
2023-02-03 14:05:29,870 [MainThread] INFO UNet3DTrainer - Training iteration [1340/60000]. Epoch [23/499]
2023-02-03 14:05:30,400 [MainThread] INFO UNet3DTrainer - Training iteration [1341/60000]. Epoch [23/499]
2023-02-03 14:05:30,935 [MainThread] INFO UNet3DTrainer - Training iteration [1342/60000]. Epoch [23/499]
2023-02-03 14:05:31,467 [MainThread] INFO UNet3DTrainer - Training iteration [1343/60000]. Epoch [23/499]
2023-02-03 14:05:31,988 [MainThread] INFO UNet3DTrainer - Training iteration [1344/60000]. Epoch [23/499]
2023-02-03 14:06:03,664 [MainThread] INFO UNet3DTrainer - Training iteration [1345/60000]. Epoch [24/499]
2023-02-03 14:06:04,081 [MainThread] INFO UNet3DTrainer - Training iteration [1346/60000]. Epoch [24/499]
2023-02-03 14:06:04,643 [MainThread] INFO UNet3DTrainer - Training iteration [1347/60000]. Epoch [24/499]
2023-02-03 14:06:05,240 [MainThread] INFO UNet3DTrainer - Training iteration [1348/60000]. Epoch [24/499]
2023-02-03 14:06:05,905 [MainThread] INFO UNet3DTrainer - Training iteration [1349/60000]. Epoch [24/499]
2023-02-03 14:06:06,460 [MainThread] INFO UNet3DTrainer - Training iteration [1350/60000]. Epoch [24/499]
2023-02-03 14:06:46,057 [MainThread] INFO UNet3DTrainer - Training iteration [1351/60000]. Epoch [24/499]
2023-02-03 14:06:46,423 [MainThread] INFO UNet3DTrainer - Training iteration [1352/60000]. Epoch [24/499]
2023-02-03 14:06:46,970 [MainThread] INFO UNet3DTrainer - Training iteration [1353/60000]. Epoch [24/499]
2023-02-03 14:06:47,507 [MainThread] INFO UNet3DTrainer - Training iteration [1354/60000]. Epoch [24/499]
2023-02-03 14:06:48,064 [MainThread] INFO UNet3DTrainer - Training iteration [1355/60000]. Epoch [24/499]
2023-02-03 14:06:48,617 [MainThread] INFO UNet3DTrainer - Training iteration [1356/60000]. Epoch [24/499]
2023-02-03 14:06:50,089 [MainThread] INFO UNet3DTrainer - Training iteration [1357/60000]. Epoch [24/499]
2023-02-03 14:06:50,600 [MainThread] INFO UNet3DTrainer - Training iteration [1358/60000]. Epoch [24/499]
2023-02-03 14:06:51,240 [MainThread] INFO UNet3DTrainer - Training iteration [1359/60000]. Epoch [24/499]
2023-02-03 14:06:51,892 [MainThread] INFO UNet3DTrainer - Training iteration [1360/60000]. Epoch [24/499]
2023-02-03 14:06:52,532 [MainThread] INFO UNet3DTrainer - Training iteration [1361/60000]. Epoch [24/499]
2023-02-03 14:06:53,109 [MainThread] INFO UNet3DTrainer - Training iteration [1362/60000]. Epoch [24/499]
2023-02-03 14:06:55,118 [MainThread] INFO UNet3DTrainer - Training iteration [1363/60000]. Epoch [24/499]
2023-02-03 14:07:14,706 [MainThread] INFO UNet3DTrainer - Training iteration [1364/60000]. Epoch [24/499]
2023-02-03 14:07:35,545 [MainThread] INFO UNet3DTrainer - Training iteration [1365/60000]. Epoch [24/499]
2023-02-03 14:07:35,936 [MainThread] INFO UNet3DTrainer - Training iteration [1366/60000]. Epoch [24/499]
2023-02-03 14:07:36,466 [MainThread] INFO UNet3DTrainer - Training iteration [1367/60000]. Epoch [24/499]
2023-02-03 14:07:37,018 [MainThread] INFO UNet3DTrainer - Training iteration [1368/60000]. Epoch [24/499]
2023-02-03 14:07:37,600 [MainThread] INFO UNet3DTrainer - Training iteration [1369/60000]. Epoch [24/499]
2023-02-03 14:08:00,811 [MainThread] INFO UNet3DTrainer - Training iteration [1370/60000]. Epoch [24/499]
2023-02-03 14:08:01,201 [MainThread] INFO UNet3DTrainer - Training iteration [1371/60000]. Epoch [24/499]
2023-02-03 14:08:01,742 [MainThread] INFO UNet3DTrainer - Training iteration [1372/60000]. Epoch [24/499]
2023-02-03 14:08:02,290 [MainThread] INFO UNet3DTrainer - Training iteration [1373/60000]. Epoch [24/499]
2023-02-03 14:08:02,848 [MainThread] INFO UNet3DTrainer - Training iteration [1374/60000]. Epoch [24/499]
2023-02-03 14:08:03,470 [MainThread] INFO UNet3DTrainer - Training iteration [1375/60000]. Epoch [24/499]
2023-02-03 14:08:31,250 [MainThread] INFO UNet3DTrainer - Training iteration [1376/60000]. Epoch [24/499]
2023-02-03 14:08:31,750 [MainThread] INFO UNet3DTrainer - Training iteration [1377/60000]. Epoch [24/499]
2023-02-03 14:08:32,304 [MainThread] INFO UNet3DTrainer - Training iteration [1378/60000]. Epoch [24/499]
2023-02-03 14:08:32,909 [MainThread] INFO UNet3DTrainer - Training iteration [1379/60000]. Epoch [24/499]
2023-02-03 14:08:33,470 [MainThread] INFO UNet3DTrainer - Training iteration [1380/60000]. Epoch [24/499]
2023-02-03 14:08:34,001 [MainThread] INFO UNet3DTrainer - Training iteration [1381/60000]. Epoch [24/499]
2023-02-03 14:08:36,462 [MainThread] INFO UNet3DTrainer - Training iteration [1382/60000]. Epoch [24/499]
2023-02-03 14:08:36,961 [MainThread] INFO UNet3DTrainer - Training iteration [1383/60000]. Epoch [24/499]
2023-02-03 14:08:37,510 [MainThread] INFO UNet3DTrainer - Training iteration [1384/60000]. Epoch [24/499]
2023-02-03 14:08:53,009 [MainThread] INFO UNet3DTrainer - Training iteration [1385/60000]. Epoch [24/499]
2023-02-03 14:08:53,387 [MainThread] INFO UNet3DTrainer - Training iteration [1386/60000]. Epoch [24/499]
2023-02-03 14:08:53,947 [MainThread] INFO UNet3DTrainer - Training iteration [1387/60000]. Epoch [24/499]
2023-02-03 14:08:54,495 [MainThread] INFO UNet3DTrainer - Training iteration [1388/60000]. Epoch [24/499]
2023-02-03 14:08:55,047 [MainThread] INFO UNet3DTrainer - Training iteration [1389/60000]. Epoch [24/499]
2023-02-03 14:08:55,600 [MainThread] INFO UNet3DTrainer - Training iteration [1390/60000]. Epoch [24/499]
2023-02-03 14:09:34,159 [MainThread] INFO UNet3DTrainer - Training iteration [1391/60000]. Epoch [24/499]
2023-02-03 14:09:34,528 [MainThread] INFO UNet3DTrainer - Training iteration [1392/60000]. Epoch [24/499]
2023-02-03 14:09:35,067 [MainThread] INFO UNet3DTrainer - Training iteration [1393/60000]. Epoch [24/499]
2023-02-03 14:09:35,592 [MainThread] INFO UNet3DTrainer - Training iteration [1394/60000]. Epoch [24/499]
2023-02-03 14:09:36,127 [MainThread] INFO UNet3DTrainer - Training iteration [1395/60000]. Epoch [24/499]
2023-02-03 14:09:36,653 [MainThread] INFO UNet3DTrainer - Training iteration [1396/60000]. Epoch [24/499]
2023-02-03 14:09:37,180 [MainThread] INFO UNet3DTrainer - Training iteration [1397/60000]. Epoch [24/499]
2023-02-03 14:09:37,715 [MainThread] INFO UNet3DTrainer - Training iteration [1398/60000]. Epoch [24/499]
2023-02-03 14:09:38,240 [MainThread] INFO UNet3DTrainer - Training iteration [1399/60000]. Epoch [24/499]
2023-02-03 14:09:38,772 [MainThread] INFO UNet3DTrainer - Training iteration [1400/60000]. Epoch [24/499]
2023-02-03 14:09:39,267 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 14:09:43,799 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 14:09:52,374 [MainThread] INFO EvalMetric - ARand: 0.6777799819454529
2023-02-03 14:09:52,414 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 14:10:00,049 [MainThread] INFO EvalMetric - ARand: 0.7394061211386729
2023-02-03 14:10:00,089 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 14:10:07,495 [MainThread] INFO EvalMetric - ARand: 0.7346970804160864
2023-02-03 14:10:07,535 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 14:10:14,963 [MainThread] INFO EvalMetric - ARand: 0.7054266069373349
2023-02-03 14:10:15,005 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 14:10:22,509 [MainThread] INFO EvalMetric - ARand: 0.7380878041879056
2023-02-03 14:10:22,549 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 14:10:29,878 [MainThread] INFO EvalMetric - ARand: 0.7317841114976421
2023-02-03 14:10:29,922 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 14:10:37,341 [MainThread] INFO EvalMetric - ARand: 0.7182179377317153
2023-02-03 14:10:37,379 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 14:10:44,700 [MainThread] INFO EvalMetric - ARand: 0.7429025519343098
2023-02-03 14:10:44,744 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 14:10:52,099 [MainThread] INFO EvalMetric - ARand: 0.6847764499668129
2023-02-03 14:10:52,141 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 14:10:59,578 [MainThread] INFO EvalMetric - ARand: 0.6811257421670974
2023-02-03 14:10:59,618 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 14:11:07,068 [MainThread] INFO EvalMetric - ARand: 0.7038338111411256
2023-02-03 14:11:07,111 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 14:11:14,617 [MainThread] INFO EvalMetric - ARand: 0.6833357176614598
2023-02-03 14:11:14,663 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 14:11:22,128 [MainThread] INFO EvalMetric - ARand: 0.6814470510947479
2023-02-03 14:11:22,170 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 14:11:29,605 [MainThread] INFO EvalMetric - ARand: 0.7812178595607163
2023-02-03 14:11:29,649 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 14:11:36,996 [MainThread] INFO EvalMetric - ARand: 0.7281717989820696
2023-02-03 14:11:37,039 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 14:11:44,431 [MainThread] INFO EvalMetric - ARand: 0.7505664417587022
2023-02-03 14:11:44,475 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 14:11:51,937 [MainThread] INFO EvalMetric - ARand: 0.6323822527539407
2023-02-03 14:11:51,980 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 14:11:59,616 [MainThread] INFO EvalMetric - ARand: 0.7415724555612508
2023-02-03 14:11:59,660 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 14:12:06,968 [MainThread] INFO EvalMetric - ARand: 0.7636178919517247
2023-02-03 14:12:07,012 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 14:12:14,359 [MainThread] INFO EvalMetric - ARand: 0.7633133194933914
2023-02-03 14:12:14,404 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 14:12:21,737 [MainThread] INFO EvalMetric - ARand: 0.7793466254850894
2023-02-03 14:12:21,784 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 14:12:27,321 [MainThread] INFO EvalMetric - ARand: 0.7777579194302843
2023-02-03 14:12:27,713 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4783860664258057. Evaluation score: 0.7239691058822971
2023-02-03 14:12:27,739 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 14:12:33,199 [MainThread] INFO EvalMetric - ARand: 0.7779174796600511
2023-02-03 14:12:33,216 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.18272839359638401. Evaluation score: 0.7779174796600511
2023-02-03 14:12:33,217 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 14:13:02,386 [MainThread] INFO UNet3DTrainer - Training iteration [1401/60000]. Epoch [25/499]
2023-02-03 14:13:02,779 [MainThread] INFO UNet3DTrainer - Training iteration [1402/60000]. Epoch [25/499]
2023-02-03 14:13:03,481 [MainThread] INFO UNet3DTrainer - Training iteration [1403/60000]. Epoch [25/499]
2023-02-03 14:13:04,039 [MainThread] INFO UNet3DTrainer - Training iteration [1404/60000]. Epoch [25/499]
2023-02-03 14:13:04,610 [MainThread] INFO UNet3DTrainer - Training iteration [1405/60000]. Epoch [25/499]
2023-02-03 14:13:05,190 [MainThread] INFO UNet3DTrainer - Training iteration [1406/60000]. Epoch [25/499]
2023-02-03 14:13:06,829 [MainThread] INFO UNet3DTrainer - Training iteration [1407/60000]. Epoch [25/499]
2023-02-03 14:13:07,255 [MainThread] INFO UNet3DTrainer - Training iteration [1408/60000]. Epoch [25/499]
2023-02-03 14:13:33,824 [MainThread] INFO UNet3DTrainer - Training iteration [1409/60000]. Epoch [25/499]
2023-02-03 14:13:34,243 [MainThread] INFO UNet3DTrainer - Training iteration [1410/60000]. Epoch [25/499]
2023-02-03 14:13:34,861 [MainThread] INFO UNet3DTrainer - Training iteration [1411/60000]. Epoch [25/499]
2023-02-03 14:13:35,453 [MainThread] INFO UNet3DTrainer - Training iteration [1412/60000]. Epoch [25/499]
2023-02-03 14:13:37,232 [MainThread] INFO UNet3DTrainer - Training iteration [1413/60000]. Epoch [25/499]
2023-02-03 14:13:37,657 [MainThread] INFO UNet3DTrainer - Training iteration [1414/60000]. Epoch [25/499]
2023-02-03 14:13:39,051 [MainThread] INFO UNet3DTrainer - Training iteration [1415/60000]. Epoch [25/499]
2023-02-03 14:13:39,537 [MainThread] INFO UNet3DTrainer - Training iteration [1416/60000]. Epoch [25/499]
2023-02-03 14:13:40,173 [MainThread] INFO UNet3DTrainer - Training iteration [1417/60000]. Epoch [25/499]
2023-02-03 14:13:40,770 [MainThread] INFO UNet3DTrainer - Training iteration [1418/60000]. Epoch [25/499]
2023-02-03 14:14:05,122 [MainThread] INFO UNet3DTrainer - Training iteration [1419/60000]. Epoch [25/499]
2023-02-03 14:14:05,540 [MainThread] INFO UNet3DTrainer - Training iteration [1420/60000]. Epoch [25/499]
2023-02-03 14:14:26,688 [MainThread] INFO UNet3DTrainer - Training iteration [1421/60000]. Epoch [25/499]
2023-02-03 14:14:27,074 [MainThread] INFO UNet3DTrainer - Training iteration [1422/60000]. Epoch [25/499]
2023-02-03 14:14:27,626 [MainThread] INFO UNet3DTrainer - Training iteration [1423/60000]. Epoch [25/499]
2023-02-03 14:14:29,426 [MainThread] INFO UNet3DTrainer - Training iteration [1424/60000]. Epoch [25/499]
2023-02-03 14:14:29,803 [MainThread] INFO UNet3DTrainer - Training iteration [1425/60000]. Epoch [25/499]
2023-02-03 14:14:30,362 [MainThread] INFO UNet3DTrainer - Training iteration [1426/60000]. Epoch [25/499]
2023-02-03 14:14:52,862 [MainThread] INFO UNet3DTrainer - Training iteration [1427/60000]. Epoch [25/499]
2023-02-03 14:14:53,255 [MainThread] INFO UNet3DTrainer - Training iteration [1428/60000]. Epoch [25/499]
2023-02-03 14:14:53,792 [MainThread] INFO UNet3DTrainer - Training iteration [1429/60000]. Epoch [25/499]
2023-02-03 14:14:54,339 [MainThread] INFO UNet3DTrainer - Training iteration [1430/60000]. Epoch [25/499]
2023-02-03 14:14:54,907 [MainThread] INFO UNet3DTrainer - Training iteration [1431/60000]. Epoch [25/499]
2023-02-03 14:14:55,471 [MainThread] INFO UNet3DTrainer - Training iteration [1432/60000]. Epoch [25/499]
2023-02-03 14:14:57,160 [MainThread] INFO UNet3DTrainer - Training iteration [1433/60000]. Epoch [25/499]
2023-02-03 14:14:57,547 [MainThread] INFO UNet3DTrainer - Training iteration [1434/60000]. Epoch [25/499]
2023-02-03 14:14:58,230 [MainThread] INFO UNet3DTrainer - Training iteration [1435/60000]. Epoch [25/499]
2023-02-03 14:14:58,879 [MainThread] INFO UNet3DTrainer - Training iteration [1436/60000]. Epoch [25/499]
2023-02-03 14:14:59,496 [MainThread] INFO UNet3DTrainer - Training iteration [1437/60000]. Epoch [25/499]
2023-02-03 14:15:00,077 [MainThread] INFO UNet3DTrainer - Training iteration [1438/60000]. Epoch [25/499]
2023-02-03 14:15:01,940 [MainThread] INFO UNet3DTrainer - Training iteration [1439/60000]. Epoch [25/499]
2023-02-03 14:15:02,430 [MainThread] INFO UNet3DTrainer - Training iteration [1440/60000]. Epoch [25/499]
2023-02-03 14:15:25,390 [MainThread] INFO UNet3DTrainer - Training iteration [1441/60000]. Epoch [25/499]
2023-02-03 14:15:25,871 [MainThread] INFO UNet3DTrainer - Training iteration [1442/60000]. Epoch [25/499]
2023-02-03 14:15:46,071 [MainThread] INFO UNet3DTrainer - Training iteration [1443/60000]. Epoch [25/499]
2023-02-03 14:15:46,459 [MainThread] INFO UNet3DTrainer - Training iteration [1444/60000]. Epoch [25/499]
2023-02-03 14:15:46,990 [MainThread] INFO UNet3DTrainer - Training iteration [1445/60000]. Epoch [25/499]
2023-02-03 14:15:47,512 [MainThread] INFO UNet3DTrainer - Training iteration [1446/60000]. Epoch [25/499]
2023-02-03 14:15:48,052 [MainThread] INFO UNet3DTrainer - Training iteration [1447/60000]. Epoch [25/499]
2023-02-03 14:15:48,604 [MainThread] INFO UNet3DTrainer - Training iteration [1448/60000]. Epoch [25/499]
2023-02-03 14:16:26,041 [MainThread] INFO UNet3DTrainer - Training iteration [1449/60000]. Epoch [25/499]
2023-02-03 14:16:26,412 [MainThread] INFO UNet3DTrainer - Training iteration [1450/60000]. Epoch [25/499]
2023-02-03 14:16:26,942 [MainThread] INFO UNet3DTrainer - Training iteration [1451/60000]. Epoch [25/499]
2023-02-03 14:16:27,481 [MainThread] INFO UNet3DTrainer - Training iteration [1452/60000]. Epoch [25/499]
2023-02-03 14:16:28,001 [MainThread] INFO UNet3DTrainer - Training iteration [1453/60000]. Epoch [25/499]
2023-02-03 14:16:28,547 [MainThread] INFO UNet3DTrainer - Training iteration [1454/60000]. Epoch [25/499]
2023-02-03 14:16:47,049 [MainThread] INFO UNet3DTrainer - Training iteration [1455/60000]. Epoch [25/499]
2023-02-03 14:16:47,416 [MainThread] INFO UNet3DTrainer - Training iteration [1456/60000]. Epoch [25/499]
2023-02-03 14:16:54,982 [MainThread] INFO UNet3DTrainer - Training iteration [1457/60000]. Epoch [26/499]
2023-02-03 14:16:55,467 [MainThread] INFO UNet3DTrainer - Training iteration [1458/60000]. Epoch [26/499]
2023-02-03 14:17:17,391 [MainThread] INFO UNet3DTrainer - Training iteration [1459/60000]. Epoch [26/499]
2023-02-03 14:17:17,777 [MainThread] INFO UNet3DTrainer - Training iteration [1460/60000]. Epoch [26/499]
2023-02-03 14:17:18,329 [MainThread] INFO UNet3DTrainer - Training iteration [1461/60000]. Epoch [26/499]
2023-02-03 14:17:18,880 [MainThread] INFO UNet3DTrainer - Training iteration [1462/60000]. Epoch [26/499]
2023-02-03 14:17:42,714 [MainThread] INFO UNet3DTrainer - Training iteration [1463/60000]. Epoch [26/499]
2023-02-03 14:17:43,106 [MainThread] INFO UNet3DTrainer - Training iteration [1464/60000]. Epoch [26/499]
2023-02-03 14:17:43,670 [MainThread] INFO UNet3DTrainer - Training iteration [1465/60000]. Epoch [26/499]
2023-02-03 14:17:44,213 [MainThread] INFO UNet3DTrainer - Training iteration [1466/60000]. Epoch [26/499]
2023-02-03 14:17:44,792 [MainThread] INFO UNet3DTrainer - Training iteration [1467/60000]. Epoch [26/499]
2023-02-03 14:17:45,360 [MainThread] INFO UNet3DTrainer - Training iteration [1468/60000]. Epoch [26/499]
2023-02-03 14:18:08,894 [MainThread] INFO UNet3DTrainer - Training iteration [1469/60000]. Epoch [26/499]
2023-02-03 14:18:09,278 [MainThread] INFO UNet3DTrainer - Training iteration [1470/60000]. Epoch [26/499]
2023-02-03 14:18:09,844 [MainThread] INFO UNet3DTrainer - Training iteration [1471/60000]. Epoch [26/499]
2023-02-03 14:18:10,415 [MainThread] INFO UNet3DTrainer - Training iteration [1472/60000]. Epoch [26/499]
2023-02-03 14:18:10,963 [MainThread] INFO UNet3DTrainer - Training iteration [1473/60000]. Epoch [26/499]
2023-02-03 14:18:11,598 [MainThread] INFO UNet3DTrainer - Training iteration [1474/60000]. Epoch [26/499]
2023-02-03 14:18:35,813 [MainThread] INFO UNet3DTrainer - Training iteration [1475/60000]. Epoch [26/499]
2023-02-03 14:18:36,224 [MainThread] INFO UNet3DTrainer - Training iteration [1476/60000]. Epoch [26/499]
2023-02-03 14:18:36,769 [MainThread] INFO UNet3DTrainer - Training iteration [1477/60000]. Epoch [26/499]
2023-02-03 14:18:37,312 [MainThread] INFO UNet3DTrainer - Training iteration [1478/60000]. Epoch [26/499]
2023-02-03 14:18:37,869 [MainThread] INFO UNet3DTrainer - Training iteration [1479/60000]. Epoch [26/499]
2023-02-03 14:18:38,460 [MainThread] INFO UNet3DTrainer - Training iteration [1480/60000]. Epoch [26/499]
2023-02-03 14:18:59,651 [MainThread] INFO UNet3DTrainer - Training iteration [1481/60000]. Epoch [26/499]
2023-02-03 14:19:00,040 [MainThread] INFO UNet3DTrainer - Training iteration [1482/60000]. Epoch [26/499]
2023-02-03 14:19:00,597 [MainThread] INFO UNet3DTrainer - Training iteration [1483/60000]. Epoch [26/499]
2023-02-03 14:19:01,150 [MainThread] INFO UNet3DTrainer - Training iteration [1484/60000]. Epoch [26/499]
2023-02-03 14:19:01,698 [MainThread] INFO UNet3DTrainer - Training iteration [1485/60000]. Epoch [26/499]
2023-02-03 14:19:02,260 [MainThread] INFO UNet3DTrainer - Training iteration [1486/60000]. Epoch [26/499]
2023-02-03 14:19:25,652 [MainThread] INFO UNet3DTrainer - Training iteration [1487/60000]. Epoch [26/499]
2023-02-03 14:19:26,033 [MainThread] INFO UNet3DTrainer - Training iteration [1488/60000]. Epoch [26/499]
2023-02-03 14:19:26,586 [MainThread] INFO UNet3DTrainer - Training iteration [1489/60000]. Epoch [26/499]
2023-02-03 14:19:27,147 [MainThread] INFO UNet3DTrainer - Training iteration [1490/60000]. Epoch [26/499]
2023-02-03 14:19:27,776 [MainThread] INFO UNet3DTrainer - Training iteration [1491/60000]. Epoch [26/499]
2023-02-03 14:19:28,409 [MainThread] INFO UNet3DTrainer - Training iteration [1492/60000]. Epoch [26/499]
2023-02-03 14:19:51,131 [MainThread] INFO UNet3DTrainer - Training iteration [1493/60000]. Epoch [26/499]
2023-02-03 14:19:51,515 [MainThread] INFO UNet3DTrainer - Training iteration [1494/60000]. Epoch [26/499]
2023-02-03 14:19:52,071 [MainThread] INFO UNet3DTrainer - Training iteration [1495/60000]. Epoch [26/499]
2023-02-03 14:19:52,621 [MainThread] INFO UNet3DTrainer - Training iteration [1496/60000]. Epoch [26/499]
2023-02-03 14:19:53,176 [MainThread] INFO UNet3DTrainer - Training iteration [1497/60000]. Epoch [26/499]
2023-02-03 14:19:53,751 [MainThread] INFO UNet3DTrainer - Training iteration [1498/60000]. Epoch [26/499]
2023-02-03 14:20:14,647 [MainThread] INFO UNet3DTrainer - Training iteration [1499/60000]. Epoch [26/499]
2023-02-03 14:20:15,025 [MainThread] INFO UNet3DTrainer - Training iteration [1500/60000]. Epoch [26/499]
2023-02-03 14:20:15,536 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 14:20:21,120 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 14:20:30,072 [MainThread] INFO EvalMetric - ARand: 0.6726253611484645
2023-02-03 14:20:30,109 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 14:20:37,684 [MainThread] INFO EvalMetric - ARand: 0.7007554175481066
2023-02-03 14:20:37,730 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 14:20:45,322 [MainThread] INFO EvalMetric - ARand: 0.7357571858061438
2023-02-03 14:20:45,359 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 14:20:52,742 [MainThread] INFO EvalMetric - ARand: 0.694065573053391
2023-02-03 14:20:52,780 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 14:21:00,224 [MainThread] INFO EvalMetric - ARand: 0.666666661469873
2023-02-03 14:21:00,268 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 14:21:07,732 [MainThread] INFO EvalMetric - ARand: 0.7156285288038047
2023-02-03 14:21:07,778 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 14:21:15,250 [MainThread] INFO EvalMetric - ARand: 0.688600685437668
2023-02-03 14:21:15,288 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 14:21:22,670 [MainThread] INFO EvalMetric - ARand: 0.7113022262347072
2023-02-03 14:21:22,712 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 14:21:29,975 [MainThread] INFO EvalMetric - ARand: 0.6806161400205036
2023-02-03 14:21:30,015 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 14:21:37,404 [MainThread] INFO EvalMetric - ARand: 0.6566224022302929
2023-02-03 14:21:37,440 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 14:21:44,826 [MainThread] INFO EvalMetric - ARand: 0.5897675377532305
2023-02-03 14:21:44,865 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 14:21:52,301 [MainThread] INFO EvalMetric - ARand: 0.6581712787901799
2023-02-03 14:21:52,345 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 14:21:59,872 [MainThread] INFO EvalMetric - ARand: 0.6537398331400829
2023-02-03 14:21:59,908 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 14:22:07,342 [MainThread] INFO EvalMetric - ARand: 0.6834901280512158
2023-02-03 14:22:07,384 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 14:22:14,795 [MainThread] INFO EvalMetric - ARand: 0.7146571874185959
2023-02-03 14:22:14,838 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 14:22:22,274 [MainThread] INFO EvalMetric - ARand: 0.7484272513467737
2023-02-03 14:22:22,315 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 14:22:29,817 [MainThread] INFO EvalMetric - ARand: 0.659634007650532
2023-02-03 14:22:29,859 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 14:22:37,341 [MainThread] INFO EvalMetric - ARand: 0.7395606630692535
2023-02-03 14:22:37,384 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 14:22:44,878 [MainThread] INFO EvalMetric - ARand: 0.681869750539198
2023-02-03 14:22:44,923 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 14:22:52,402 [MainThread] INFO EvalMetric - ARand: 0.6805345513638855
2023-02-03 14:22:52,447 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 14:22:59,896 [MainThread] INFO EvalMetric - ARand: 0.783188343450187
2023-02-03 14:22:59,939 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 14:23:05,611 [MainThread] INFO EvalMetric - ARand: 0.7125562547862746
2023-02-03 14:23:05,998 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4769759068543884. Evaluation score: 0.6919585243869331
2023-02-03 14:23:06,022 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 14:23:13,184 [MainThread] INFO EvalMetric - ARand: 0.6738670789780421
2023-02-03 14:23:13,203 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.18027672950517049. Evaluation score: 0.6738670789780421
2023-02-03 14:23:13,205 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 14:23:14,242 [MainThread] INFO UNet3DTrainer - Training iteration [1501/60000]. Epoch [26/499]
2023-02-03 14:23:14,604 [MainThread] INFO UNet3DTrainer - Training iteration [1502/60000]. Epoch [26/499]
2023-02-03 14:23:15,136 [MainThread] INFO UNet3DTrainer - Training iteration [1503/60000]. Epoch [26/499]
2023-02-03 14:23:15,677 [MainThread] INFO UNet3DTrainer - Training iteration [1504/60000]. Epoch [26/499]
2023-02-03 14:23:16,205 [MainThread] INFO UNet3DTrainer - Training iteration [1505/60000]. Epoch [26/499]
2023-02-03 14:23:16,721 [MainThread] INFO UNet3DTrainer - Training iteration [1506/60000]. Epoch [26/499]
2023-02-03 14:23:17,254 [MainThread] INFO UNet3DTrainer - Training iteration [1507/60000]. Epoch [26/499]
2023-02-03 14:23:17,781 [MainThread] INFO UNet3DTrainer - Training iteration [1508/60000]. Epoch [26/499]
2023-02-03 14:23:18,314 [MainThread] INFO UNet3DTrainer - Training iteration [1509/60000]. Epoch [26/499]
2023-02-03 14:23:18,840 [MainThread] INFO UNet3DTrainer - Training iteration [1510/60000]. Epoch [26/499]
2023-02-03 14:23:19,366 [MainThread] INFO UNet3DTrainer - Training iteration [1511/60000]. Epoch [26/499]
2023-02-03 14:23:19,888 [MainThread] INFO UNet3DTrainer - Training iteration [1512/60000]. Epoch [26/499]
2023-02-03 14:23:28,004 [MainThread] INFO UNet3DTrainer - Training iteration [1513/60000]. Epoch [27/499]
2023-02-03 14:24:10,624 [MainThread] INFO UNet3DTrainer - Training iteration [1514/60000]. Epoch [27/499]
2023-02-03 14:24:10,998 [MainThread] INFO UNet3DTrainer - Training iteration [1515/60000]. Epoch [27/499]
2023-02-03 14:24:11,538 [MainThread] INFO UNet3DTrainer - Training iteration [1516/60000]. Epoch [27/499]
2023-02-03 14:24:12,085 [MainThread] INFO UNet3DTrainer - Training iteration [1517/60000]. Epoch [27/499]
2023-02-03 14:24:12,644 [MainThread] INFO UNet3DTrainer - Training iteration [1518/60000]. Epoch [27/499]
2023-02-03 14:24:13,240 [MainThread] INFO UNet3DTrainer - Training iteration [1519/60000]. Epoch [27/499]
2023-02-03 14:24:36,541 [MainThread] INFO UNet3DTrainer - Training iteration [1520/60000]. Epoch [27/499]
2023-02-03 14:24:36,939 [MainThread] INFO UNet3DTrainer - Training iteration [1521/60000]. Epoch [27/499]
2023-02-03 14:24:37,482 [MainThread] INFO UNet3DTrainer - Training iteration [1522/60000]. Epoch [27/499]
2023-02-03 14:24:38,022 [MainThread] INFO UNet3DTrainer - Training iteration [1523/60000]. Epoch [27/499]
2023-02-03 14:24:38,583 [MainThread] INFO UNet3DTrainer - Training iteration [1524/60000]. Epoch [27/499]
2023-02-03 14:24:39,162 [MainThread] INFO UNet3DTrainer - Training iteration [1525/60000]. Epoch [27/499]
2023-02-03 14:25:04,233 [MainThread] INFO UNet3DTrainer - Training iteration [1526/60000]. Epoch [27/499]
2023-02-03 14:25:04,621 [MainThread] INFO UNet3DTrainer - Training iteration [1527/60000]. Epoch [27/499]
2023-02-03 14:25:05,185 [MainThread] INFO UNet3DTrainer - Training iteration [1528/60000]. Epoch [27/499]
2023-02-03 14:25:05,750 [MainThread] INFO UNet3DTrainer - Training iteration [1529/60000]. Epoch [27/499]
2023-02-03 14:25:06,407 [MainThread] INFO UNet3DTrainer - Training iteration [1530/60000]. Epoch [27/499]
2023-02-03 14:25:06,973 [MainThread] INFO UNet3DTrainer - Training iteration [1531/60000]. Epoch [27/499]
2023-02-03 14:25:09,052 [MainThread] INFO UNet3DTrainer - Training iteration [1532/60000]. Epoch [27/499]
2023-02-03 14:25:09,480 [MainThread] INFO UNet3DTrainer - Training iteration [1533/60000]. Epoch [27/499]
2023-02-03 14:25:29,800 [MainThread] INFO UNet3DTrainer - Training iteration [1534/60000]. Epoch [27/499]
2023-02-03 14:25:30,188 [MainThread] INFO UNet3DTrainer - Training iteration [1535/60000]. Epoch [27/499]
2023-02-03 14:25:30,744 [MainThread] INFO UNet3DTrainer - Training iteration [1536/60000]. Epoch [27/499]
2023-02-03 14:25:31,320 [MainThread] INFO UNet3DTrainer - Training iteration [1537/60000]. Epoch [27/499]
2023-02-03 14:25:36,133 [MainThread] INFO UNet3DTrainer - Training iteration [1538/60000]. Epoch [27/499]
2023-02-03 14:25:51,836 [MainThread] INFO UNet3DTrainer - Training iteration [1539/60000]. Epoch [27/499]
2023-02-03 14:25:52,210 [MainThread] INFO UNet3DTrainer - Training iteration [1540/60000]. Epoch [27/499]
2023-02-03 14:25:52,744 [MainThread] INFO UNet3DTrainer - Training iteration [1541/60000]. Epoch [27/499]
2023-02-03 14:25:53,297 [MainThread] INFO UNet3DTrainer - Training iteration [1542/60000]. Epoch [27/499]
2023-02-03 14:25:53,852 [MainThread] INFO UNet3DTrainer - Training iteration [1543/60000]. Epoch [27/499]
2023-02-03 14:25:54,430 [MainThread] INFO UNet3DTrainer - Training iteration [1544/60000]. Epoch [27/499]
2023-02-03 14:26:17,520 [MainThread] INFO UNet3DTrainer - Training iteration [1545/60000]. Epoch [27/499]
2023-02-03 14:26:17,922 [MainThread] INFO UNet3DTrainer - Training iteration [1546/60000]. Epoch [27/499]
2023-02-03 14:26:18,470 [MainThread] INFO UNet3DTrainer - Training iteration [1547/60000]. Epoch [27/499]
2023-02-03 14:26:19,050 [MainThread] INFO UNet3DTrainer - Training iteration [1548/60000]. Epoch [27/499]
2023-02-03 14:26:19,652 [MainThread] INFO UNet3DTrainer - Training iteration [1549/60000]. Epoch [27/499]
2023-02-03 14:26:20,287 [MainThread] INFO UNet3DTrainer - Training iteration [1550/60000]. Epoch [27/499]
2023-02-03 14:26:40,693 [MainThread] INFO UNet3DTrainer - Training iteration [1551/60000]. Epoch [27/499]
2023-02-03 14:26:41,067 [MainThread] INFO UNet3DTrainer - Training iteration [1552/60000]. Epoch [27/499]
2023-02-03 14:26:41,602 [MainThread] INFO UNet3DTrainer - Training iteration [1553/60000]. Epoch [27/499]
2023-02-03 14:26:42,155 [MainThread] INFO UNet3DTrainer - Training iteration [1554/60000]. Epoch [27/499]
2023-02-03 14:26:42,702 [MainThread] INFO UNet3DTrainer - Training iteration [1555/60000]. Epoch [27/499]
2023-02-03 14:26:43,312 [MainThread] INFO UNet3DTrainer - Training iteration [1556/60000]. Epoch [27/499]
2023-02-03 14:27:04,149 [MainThread] INFO UNet3DTrainer - Training iteration [1557/60000]. Epoch [27/499]
2023-02-03 14:27:04,533 [MainThread] INFO UNet3DTrainer - Training iteration [1558/60000]. Epoch [27/499]
2023-02-03 14:27:05,072 [MainThread] INFO UNet3DTrainer - Training iteration [1559/60000]. Epoch [27/499]
2023-02-03 14:27:05,604 [MainThread] INFO UNet3DTrainer - Training iteration [1560/60000]. Epoch [27/499]
2023-02-03 14:27:06,149 [MainThread] INFO UNet3DTrainer - Training iteration [1561/60000]. Epoch [27/499]
2023-02-03 14:27:06,695 [MainThread] INFO UNet3DTrainer - Training iteration [1562/60000]. Epoch [27/499]
2023-02-03 14:27:07,389 [MainThread] INFO UNet3DTrainer - Training iteration [1563/60000]. Epoch [27/499]
2023-02-03 14:27:07,754 [MainThread] INFO UNet3DTrainer - Training iteration [1564/60000]. Epoch [27/499]
2023-02-03 14:27:23,112 [MainThread] INFO UNet3DTrainer - Training iteration [1565/60000]. Epoch [27/499]
2023-02-03 14:27:23,501 [MainThread] INFO UNet3DTrainer - Training iteration [1566/60000]. Epoch [27/499]
2023-02-03 14:27:24,029 [MainThread] INFO UNet3DTrainer - Training iteration [1567/60000]. Epoch [27/499]
2023-02-03 14:27:24,548 [MainThread] INFO UNet3DTrainer - Training iteration [1568/60000]. Epoch [27/499]
2023-02-03 14:27:58,073 [MainThread] INFO UNet3DTrainer - Training iteration [1569/60000]. Epoch [28/499]
2023-02-03 14:27:58,600 [MainThread] INFO UNet3DTrainer - Training iteration [1570/60000]. Epoch [28/499]
2023-02-03 14:27:59,231 [MainThread] INFO UNet3DTrainer - Training iteration [1571/60000]. Epoch [28/499]
2023-02-03 14:27:59,834 [MainThread] INFO UNet3DTrainer - Training iteration [1572/60000]. Epoch [28/499]
2023-02-03 14:28:00,390 [MainThread] INFO UNet3DTrainer - Training iteration [1573/60000]. Epoch [28/499]
2023-02-03 14:28:00,959 [MainThread] INFO UNet3DTrainer - Training iteration [1574/60000]. Epoch [28/499]
2023-02-03 14:28:03,262 [MainThread] INFO UNet3DTrainer - Training iteration [1575/60000]. Epoch [28/499]
2023-02-03 14:28:03,807 [MainThread] INFO UNet3DTrainer - Training iteration [1576/60000]. Epoch [28/499]
2023-02-03 14:28:04,413 [MainThread] INFO UNet3DTrainer - Training iteration [1577/60000]. Epoch [28/499]
2023-02-03 14:28:05,050 [MainThread] INFO UNet3DTrainer - Training iteration [1578/60000]. Epoch [28/499]
2023-02-03 14:28:05,640 [MainThread] INFO UNet3DTrainer - Training iteration [1579/60000]. Epoch [28/499]
2023-02-03 14:28:06,346 [MainThread] INFO UNet3DTrainer - Training iteration [1580/60000]. Epoch [28/499]
2023-02-03 14:28:08,790 [MainThread] INFO UNet3DTrainer - Training iteration [1581/60000]. Epoch [28/499]
2023-02-03 14:28:31,834 [MainThread] INFO UNet3DTrainer - Training iteration [1582/60000]. Epoch [28/499]
2023-02-03 14:28:32,242 [MainThread] INFO UNet3DTrainer - Training iteration [1583/60000]. Epoch [28/499]
2023-02-03 14:28:51,875 [MainThread] INFO UNet3DTrainer - Training iteration [1584/60000]. Epoch [28/499]
2023-02-03 14:28:52,262 [MainThread] INFO UNet3DTrainer - Training iteration [1585/60000]. Epoch [28/499]
2023-02-03 14:28:52,812 [MainThread] INFO UNet3DTrainer - Training iteration [1586/60000]. Epoch [28/499]
2023-02-03 14:28:53,373 [MainThread] INFO UNet3DTrainer - Training iteration [1587/60000]. Epoch [28/499]
2023-02-03 14:28:53,947 [MainThread] INFO UNet3DTrainer - Training iteration [1588/60000]. Epoch [28/499]
2023-02-03 14:28:56,280 [MainThread] INFO UNet3DTrainer - Training iteration [1589/60000]. Epoch [28/499]
2023-02-03 14:29:17,646 [MainThread] INFO UNet3DTrainer - Training iteration [1590/60000]. Epoch [28/499]
2023-02-03 14:29:18,038 [MainThread] INFO UNet3DTrainer - Training iteration [1591/60000]. Epoch [28/499]
2023-02-03 14:29:18,585 [MainThread] INFO UNet3DTrainer - Training iteration [1592/60000]. Epoch [28/499]
2023-02-03 14:29:19,164 [MainThread] INFO UNet3DTrainer - Training iteration [1593/60000]. Epoch [28/499]
2023-02-03 14:29:19,701 [MainThread] INFO UNet3DTrainer - Training iteration [1594/60000]. Epoch [28/499]
2023-02-03 14:29:20,364 [MainThread] INFO UNet3DTrainer - Training iteration [1595/60000]. Epoch [28/499]
2023-02-03 14:29:22,220 [MainThread] INFO UNet3DTrainer - Training iteration [1596/60000]. Epoch [28/499]
2023-02-03 14:29:22,682 [MainThread] INFO UNet3DTrainer - Training iteration [1597/60000]. Epoch [28/499]
2023-02-03 14:29:43,052 [MainThread] INFO UNet3DTrainer - Training iteration [1598/60000]. Epoch [28/499]
2023-02-03 14:29:43,435 [MainThread] INFO UNet3DTrainer - Training iteration [1599/60000]. Epoch [28/499]
2023-02-03 14:29:44,004 [MainThread] INFO UNet3DTrainer - Training iteration [1600/60000]. Epoch [28/499]
2023-02-03 14:29:44,606 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 14:29:51,065 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 14:30:03,276 [MainThread] INFO EvalMetric - ARand: 0.6612409213003358
2023-02-03 14:30:03,318 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 14:30:11,473 [MainThread] INFO EvalMetric - ARand: 0.68655196622547
2023-02-03 14:30:11,521 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 14:30:19,658 [MainThread] INFO EvalMetric - ARand: 0.7247606527263444
2023-02-03 14:30:19,698 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 14:30:27,153 [MainThread] INFO EvalMetric - ARand: 0.6646094331251606
2023-02-03 14:30:27,203 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 14:30:34,630 [MainThread] INFO EvalMetric - ARand: 0.6531362153092352
2023-02-03 14:30:34,669 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 14:30:42,073 [MainThread] INFO EvalMetric - ARand: 0.7404560533251002
2023-02-03 14:30:42,114 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 14:30:56,935 [MainThread] INFO EvalMetric - ARand: 0.7104164867869336
2023-02-03 14:30:56,972 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 14:31:04,330 [MainThread] INFO EvalMetric - ARand: 0.7367687522117666
2023-02-03 14:31:04,368 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 14:31:11,805 [MainThread] INFO EvalMetric - ARand: 0.6704663205769308
2023-02-03 14:31:11,847 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 14:31:19,257 [MainThread] INFO EvalMetric - ARand: 0.5714504578474564
2023-02-03 14:31:19,299 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 14:31:26,708 [MainThread] INFO EvalMetric - ARand: 0.6035392356954769
2023-02-03 14:31:26,746 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 14:31:34,130 [MainThread] INFO EvalMetric - ARand: 0.6435819669943326
2023-02-03 14:31:34,176 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 14:31:41,579 [MainThread] INFO EvalMetric - ARand: 0.6975407802262956
2023-02-03 14:31:41,616 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 14:31:48,990 [MainThread] INFO EvalMetric - ARand: 0.7185402987412064
2023-02-03 14:31:49,028 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 14:31:56,577 [MainThread] INFO EvalMetric - ARand: 0.6966355786553188
2023-02-03 14:31:56,627 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 14:32:04,177 [MainThread] INFO EvalMetric - ARand: 0.6426832847417638
2023-02-03 14:32:04,224 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 14:32:11,653 [MainThread] INFO EvalMetric - ARand: 0.6552368597612575
2023-02-03 14:32:11,698 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 14:32:19,213 [MainThread] INFO EvalMetric - ARand: 0.7098605059201362
2023-02-03 14:32:19,259 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 14:32:26,719 [MainThread] INFO EvalMetric - ARand: 0.7041175505084654
2023-02-03 14:32:26,763 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 14:32:34,205 [MainThread] INFO EvalMetric - ARand: 0.7899749739550043
2023-02-03 14:32:34,249 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 14:32:39,919 [MainThread] INFO EvalMetric - ARand: 0.7487552986673452
2023-02-03 14:32:40,308 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4766894373400459. Evaluation score: 0.6854691369125252
2023-02-03 14:32:40,330 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 14:32:47,544 [MainThread] INFO EvalMetric - ARand: 0.7782141989208103
2023-02-03 14:32:47,564 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.1741839055903256. Evaluation score: 0.7782141989208103
2023-02-03 14:32:47,565 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 14:32:48,606 [MainThread] INFO UNet3DTrainer - Training iteration [1601/60000]. Epoch [28/499]
2023-02-03 14:32:48,988 [MainThread] INFO UNet3DTrainer - Training iteration [1602/60000]. Epoch [28/499]
2023-02-03 14:32:49,519 [MainThread] INFO UNet3DTrainer - Training iteration [1603/60000]. Epoch [28/499]
2023-02-03 14:32:50,068 [MainThread] INFO UNet3DTrainer - Training iteration [1604/60000]. Epoch [28/499]
2023-02-03 14:32:50,598 [MainThread] INFO UNet3DTrainer - Training iteration [1605/60000]. Epoch [28/499]
2023-02-03 14:32:51,166 [MainThread] INFO UNet3DTrainer - Training iteration [1606/60000]. Epoch [28/499]
2023-02-03 14:32:51,772 [MainThread] INFO UNet3DTrainer - Training iteration [1607/60000]. Epoch [28/499]
2023-02-03 14:32:52,364 [MainThread] INFO UNet3DTrainer - Training iteration [1608/60000]. Epoch [28/499]
2023-02-03 14:32:52,953 [MainThread] INFO UNet3DTrainer - Training iteration [1609/60000]. Epoch [28/499]
2023-02-03 14:32:53,618 [MainThread] INFO UNet3DTrainer - Training iteration [1610/60000]. Epoch [28/499]
2023-02-03 14:32:54,167 [MainThread] INFO UNet3DTrainer - Training iteration [1611/60000]. Epoch [28/499]
2023-02-03 14:32:54,740 [MainThread] INFO UNet3DTrainer - Training iteration [1612/60000]. Epoch [28/499]
2023-02-03 14:33:14,403 [MainThread] INFO UNet3DTrainer - Training iteration [1613/60000]. Epoch [28/499]
2023-02-03 14:33:52,036 [MainThread] INFO UNet3DTrainer - Training iteration [1614/60000]. Epoch [28/499]
2023-02-03 14:33:52,396 [MainThread] INFO UNet3DTrainer - Training iteration [1615/60000]. Epoch [28/499]
2023-02-03 14:33:52,932 [MainThread] INFO UNet3DTrainer - Training iteration [1616/60000]. Epoch [28/499]
2023-02-03 14:33:53,464 [MainThread] INFO UNet3DTrainer - Training iteration [1617/60000]. Epoch [28/499]
2023-02-03 14:33:53,996 [MainThread] INFO UNet3DTrainer - Training iteration [1618/60000]. Epoch [28/499]
2023-02-03 14:33:54,546 [MainThread] INFO UNet3DTrainer - Training iteration [1619/60000]. Epoch [28/499]
2023-02-03 14:33:55,069 [MainThread] INFO UNet3DTrainer - Training iteration [1620/60000]. Epoch [28/499]
2023-02-03 14:33:55,603 [MainThread] INFO UNet3DTrainer - Training iteration [1621/60000]. Epoch [28/499]
2023-02-03 14:33:56,136 [MainThread] INFO UNet3DTrainer - Training iteration [1622/60000]. Epoch [28/499]
2023-02-03 14:33:56,655 [MainThread] INFO UNet3DTrainer - Training iteration [1623/60000]. Epoch [28/499]
2023-02-03 14:33:57,180 [MainThread] INFO UNet3DTrainer - Training iteration [1624/60000]. Epoch [28/499]
2023-02-03 14:34:27,049 [MainThread] INFO UNet3DTrainer - Training iteration [1625/60000]. Epoch [29/499]
2023-02-03 14:34:27,456 [MainThread] INFO UNet3DTrainer - Training iteration [1626/60000]. Epoch [29/499]
2023-02-03 14:34:28,004 [MainThread] INFO UNet3DTrainer - Training iteration [1627/60000]. Epoch [29/499]
2023-02-03 14:34:28,545 [MainThread] INFO UNet3DTrainer - Training iteration [1628/60000]. Epoch [29/499]
2023-02-03 14:34:29,107 [MainThread] INFO UNet3DTrainer - Training iteration [1629/60000]. Epoch [29/499]
2023-02-03 14:34:29,670 [MainThread] INFO UNet3DTrainer - Training iteration [1630/60000]. Epoch [29/499]
2023-02-03 14:34:31,351 [MainThread] INFO UNet3DTrainer - Training iteration [1631/60000]. Epoch [29/499]
2023-02-03 14:34:31,760 [MainThread] INFO UNet3DTrainer - Training iteration [1632/60000]. Epoch [29/499]
2023-02-03 14:34:32,365 [MainThread] INFO UNet3DTrainer - Training iteration [1633/60000]. Epoch [29/499]
2023-02-03 14:34:32,960 [MainThread] INFO UNet3DTrainer - Training iteration [1634/60000]. Epoch [29/499]
2023-02-03 14:34:52,070 [MainThread] INFO UNet3DTrainer - Training iteration [1635/60000]. Epoch [29/499]
2023-02-03 14:34:52,450 [MainThread] INFO UNet3DTrainer - Training iteration [1636/60000]. Epoch [29/499]
2023-02-03 14:34:52,999 [MainThread] INFO UNet3DTrainer - Training iteration [1637/60000]. Epoch [29/499]
2023-02-03 14:34:53,554 [MainThread] INFO UNet3DTrainer - Training iteration [1638/60000]. Epoch [29/499]
2023-02-03 14:34:54,100 [MainThread] INFO UNet3DTrainer - Training iteration [1639/60000]. Epoch [29/499]
2023-02-03 14:34:54,781 [MainThread] INFO UNet3DTrainer - Training iteration [1640/60000]. Epoch [29/499]
2023-02-03 14:35:18,686 [MainThread] INFO UNet3DTrainer - Training iteration [1641/60000]. Epoch [29/499]
2023-02-03 14:35:19,082 [MainThread] INFO UNet3DTrainer - Training iteration [1642/60000]. Epoch [29/499]
2023-02-03 14:35:19,631 [MainThread] INFO UNet3DTrainer - Training iteration [1643/60000]. Epoch [29/499]
2023-02-03 14:35:20,187 [MainThread] INFO UNet3DTrainer - Training iteration [1644/60000]. Epoch [29/499]
2023-02-03 14:35:20,810 [MainThread] INFO UNet3DTrainer - Training iteration [1645/60000]. Epoch [29/499]
2023-02-03 14:35:21,426 [MainThread] INFO UNet3DTrainer - Training iteration [1646/60000]. Epoch [29/499]
2023-02-03 14:35:23,983 [MainThread] INFO UNet3DTrainer - Training iteration [1647/60000]. Epoch [29/499]
2023-02-03 14:35:56,341 [MainThread] INFO UNet3DTrainer - Training iteration [1648/60000]. Epoch [29/499]
2023-02-03 14:35:56,695 [MainThread] INFO UNet3DTrainer - Training iteration [1649/60000]. Epoch [29/499]
2023-02-03 14:35:57,219 [MainThread] INFO UNet3DTrainer - Training iteration [1650/60000]. Epoch [29/499]
2023-02-03 14:35:57,773 [MainThread] INFO UNet3DTrainer - Training iteration [1651/60000]. Epoch [29/499]
2023-02-03 14:35:58,326 [MainThread] INFO UNet3DTrainer - Training iteration [1652/60000]. Epoch [29/499]
2023-02-03 14:35:58,913 [MainThread] INFO UNet3DTrainer - Training iteration [1653/60000]. Epoch [29/499]
2023-02-03 14:36:18,979 [MainThread] INFO UNet3DTrainer - Training iteration [1654/60000]. Epoch [29/499]
2023-02-03 14:36:19,375 [MainThread] INFO UNet3DTrainer - Training iteration [1655/60000]. Epoch [29/499]
2023-02-03 14:36:19,920 [MainThread] INFO UNet3DTrainer - Training iteration [1656/60000]. Epoch [29/499]
2023-02-03 14:36:20,459 [MainThread] INFO UNet3DTrainer - Training iteration [1657/60000]. Epoch [29/499]
2023-02-03 14:36:21,015 [MainThread] INFO UNet3DTrainer - Training iteration [1658/60000]. Epoch [29/499]
2023-02-03 14:36:21,557 [MainThread] INFO UNet3DTrainer - Training iteration [1659/60000]. Epoch [29/499]
2023-02-03 14:36:23,148 [MainThread] INFO UNet3DTrainer - Training iteration [1660/60000]. Epoch [29/499]
2023-02-03 14:36:23,640 [MainThread] INFO UNet3DTrainer - Training iteration [1661/60000]. Epoch [29/499]
2023-02-03 14:36:40,374 [MainThread] INFO UNet3DTrainer - Training iteration [1662/60000]. Epoch [29/499]
2023-02-03 14:36:40,784 [MainThread] INFO UNet3DTrainer - Training iteration [1663/60000]. Epoch [29/499]
2023-02-03 14:36:41,706 [MainThread] INFO UNet3DTrainer - Training iteration [1664/60000]. Epoch [29/499]
2023-02-03 14:36:42,069 [MainThread] INFO UNet3DTrainer - Training iteration [1665/60000]. Epoch [29/499]
2023-02-03 14:36:47,407 [MainThread] INFO UNet3DTrainer - Training iteration [1666/60000]. Epoch [29/499]
2023-02-03 14:36:47,847 [MainThread] INFO UNet3DTrainer - Training iteration [1667/60000]. Epoch [29/499]
2023-02-03 14:37:30,134 [MainThread] INFO UNet3DTrainer - Training iteration [1668/60000]. Epoch [29/499]
2023-02-03 14:37:30,514 [MainThread] INFO UNet3DTrainer - Training iteration [1669/60000]. Epoch [29/499]
2023-02-03 14:37:31,051 [MainThread] INFO UNet3DTrainer - Training iteration [1670/60000]. Epoch [29/499]
2023-02-03 14:37:31,578 [MainThread] INFO UNet3DTrainer - Training iteration [1671/60000]. Epoch [29/499]
2023-02-03 14:37:32,108 [MainThread] INFO UNet3DTrainer - Training iteration [1672/60000]. Epoch [29/499]
2023-02-03 14:37:32,646 [MainThread] INFO UNet3DTrainer - Training iteration [1673/60000]. Epoch [29/499]
2023-02-03 14:37:51,631 [MainThread] INFO UNet3DTrainer - Training iteration [1674/60000]. Epoch [29/499]
2023-02-03 14:37:52,004 [MainThread] INFO UNet3DTrainer - Training iteration [1675/60000]. Epoch [29/499]
2023-02-03 14:37:52,539 [MainThread] INFO UNet3DTrainer - Training iteration [1676/60000]. Epoch [29/499]
2023-02-03 14:37:53,068 [MainThread] INFO UNet3DTrainer - Training iteration [1677/60000]. Epoch [29/499]
2023-02-03 14:37:58,323 [MainThread] INFO UNet3DTrainer - Training iteration [1678/60000]. Epoch [29/499]
2023-02-03 14:37:58,702 [MainThread] INFO UNet3DTrainer - Training iteration [1679/60000]. Epoch [29/499]
2023-02-03 14:38:29,990 [MainThread] INFO UNet3DTrainer - Training iteration [1680/60000]. Epoch [29/499]
2023-02-03 14:38:37,519 [MainThread] INFO UNet3DTrainer - Training iteration [1681/60000]. Epoch [30/499]
2023-02-03 14:38:38,011 [MainThread] INFO UNet3DTrainer - Training iteration [1682/60000]. Epoch [30/499]
2023-02-03 14:38:59,375 [MainThread] INFO UNet3DTrainer - Training iteration [1683/60000]. Epoch [30/499]
2023-02-03 14:38:59,745 [MainThread] INFO UNet3DTrainer - Training iteration [1684/60000]. Epoch [30/499]
2023-02-03 14:39:00,298 [MainThread] INFO UNet3DTrainer - Training iteration [1685/60000]. Epoch [30/499]
2023-02-03 14:39:00,852 [MainThread] INFO UNet3DTrainer - Training iteration [1686/60000]. Epoch [30/499]
2023-02-03 14:39:01,520 [MainThread] INFO UNet3DTrainer - Training iteration [1687/60000]. Epoch [30/499]
2023-02-03 14:39:03,808 [MainThread] INFO UNet3DTrainer - Training iteration [1688/60000]. Epoch [30/499]
2023-02-03 14:39:04,364 [MainThread] INFO UNet3DTrainer - Training iteration [1689/60000]. Epoch [30/499]
2023-02-03 14:39:05,549 [MainThread] INFO UNet3DTrainer - Training iteration [1690/60000]. Epoch [30/499]
2023-02-03 14:39:06,020 [MainThread] INFO UNet3DTrainer - Training iteration [1691/60000]. Epoch [30/499]
2023-02-03 14:39:06,703 [MainThread] INFO UNet3DTrainer - Training iteration [1692/60000]. Epoch [30/499]
2023-02-03 14:39:08,548 [MainThread] INFO UNet3DTrainer - Training iteration [1693/60000]. Epoch [30/499]
2023-02-03 14:39:08,985 [MainThread] INFO UNet3DTrainer - Training iteration [1694/60000]. Epoch [30/499]
2023-02-03 14:39:09,630 [MainThread] INFO UNet3DTrainer - Training iteration [1695/60000]. Epoch [30/499]
2023-02-03 14:39:10,410 [MainThread] INFO UNet3DTrainer - Training iteration [1696/60000]. Epoch [30/499]
2023-02-03 14:39:51,252 [MainThread] INFO UNet3DTrainer - Training iteration [1697/60000]. Epoch [30/499]
2023-02-03 14:39:51,624 [MainThread] INFO UNet3DTrainer - Training iteration [1698/60000]. Epoch [30/499]
2023-02-03 14:39:52,173 [MainThread] INFO UNet3DTrainer - Training iteration [1699/60000]. Epoch [30/499]
2023-02-03 14:39:52,710 [MainThread] INFO UNet3DTrainer - Training iteration [1700/60000]. Epoch [30/499]
2023-02-03 14:39:53,237 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 14:39:59,790 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 14:40:09,405 [MainThread] INFO EvalMetric - ARand: 0.6437613254503307
2023-02-03 14:40:09,446 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 14:40:16,777 [MainThread] INFO EvalMetric - ARand: 0.6586177198825865
2023-02-03 14:40:16,831 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 14:40:24,245 [MainThread] INFO EvalMetric - ARand: 0.6928467710192887
2023-02-03 14:40:24,292 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 14:40:31,794 [MainThread] INFO EvalMetric - ARand: 0.5954390001537366
2023-02-03 14:40:31,832 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 14:40:39,271 [MainThread] INFO EvalMetric - ARand: 0.6499295315005468
2023-02-03 14:40:39,310 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 14:40:46,733 [MainThread] INFO EvalMetric - ARand: 0.6892811569913447
2023-02-03 14:40:46,776 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 14:40:54,254 [MainThread] INFO EvalMetric - ARand: 0.6695608361821574
2023-02-03 14:40:54,291 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 14:41:01,661 [MainThread] INFO EvalMetric - ARand: 0.6478182851745771
2023-02-03 14:41:01,707 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 14:41:09,032 [MainThread] INFO EvalMetric - ARand: 0.6833178730084281
2023-02-03 14:41:09,072 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 14:41:16,507 [MainThread] INFO EvalMetric - ARand: 0.6285272860808613
2023-02-03 14:41:16,547 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 14:41:23,975 [MainThread] INFO EvalMetric - ARand: 0.6179954709963709
2023-02-03 14:41:24,012 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 14:41:31,478 [MainThread] INFO EvalMetric - ARand: 0.6604328414614695
2023-02-03 14:41:31,521 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 14:41:38,953 [MainThread] INFO EvalMetric - ARand: 0.6251316261294764
2023-02-03 14:41:38,999 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 14:41:46,450 [MainThread] INFO EvalMetric - ARand: 0.7109207681022798
2023-02-03 14:41:46,492 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 14:41:53,960 [MainThread] INFO EvalMetric - ARand: 0.6599507493839782
2023-02-03 14:41:54,004 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 14:42:01,620 [MainThread] INFO EvalMetric - ARand: 0.6562536412359745
2023-02-03 14:42:01,666 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 14:42:09,246 [MainThread] INFO EvalMetric - ARand: 0.6195738642341726
2023-02-03 14:42:09,290 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 14:42:16,773 [MainThread] INFO EvalMetric - ARand: 0.6614760261881433
2023-02-03 14:42:16,818 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 14:42:24,378 [MainThread] INFO EvalMetric - ARand: 0.6953800475004273
2023-02-03 14:42:24,423 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 14:42:31,979 [MainThread] INFO EvalMetric - ARand: 0.6602049412980957
2023-02-03 14:42:32,025 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 14:42:39,582 [MainThread] INFO EvalMetric - ARand: 0.7119482606854055
2023-02-03 14:42:39,626 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 14:42:45,278 [MainThread] INFO EvalMetric - ARand: 0.7130009694750284
2023-02-03 14:42:45,668 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4774727026621501. Evaluation score: 0.6608330459662494
2023-02-03 14:42:45,694 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 14:42:53,071 [MainThread] INFO EvalMetric - ARand: 0.7805099409339831
2023-02-03 14:42:53,090 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.179202850908041. Evaluation score: 0.7805099409339831
2023-02-03 14:42:53,092 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 14:42:54,158 [MainThread] INFO UNet3DTrainer - Training iteration [1701/60000]. Epoch [30/499]
2023-02-03 14:42:54,545 [MainThread] INFO UNet3DTrainer - Training iteration [1702/60000]. Epoch [30/499]
2023-02-03 14:42:55,074 [MainThread] INFO UNet3DTrainer - Training iteration [1703/60000]. Epoch [30/499]
2023-02-03 14:42:55,603 [MainThread] INFO UNet3DTrainer - Training iteration [1704/60000]. Epoch [30/499]
2023-02-03 14:42:56,156 [MainThread] INFO UNet3DTrainer - Training iteration [1705/60000]. Epoch [30/499]
2023-02-03 14:42:56,728 [MainThread] INFO UNet3DTrainer - Training iteration [1706/60000]. Epoch [30/499]
2023-02-03 14:42:57,350 [MainThread] INFO UNet3DTrainer - Training iteration [1707/60000]. Epoch [30/499]
2023-02-03 14:42:57,930 [MainThread] INFO UNet3DTrainer - Training iteration [1708/60000]. Epoch [30/499]
2023-02-03 14:42:58,530 [MainThread] INFO UNet3DTrainer - Training iteration [1709/60000]. Epoch [30/499]
2023-02-03 14:42:59,112 [MainThread] INFO UNet3DTrainer - Training iteration [1710/60000]. Epoch [30/499]
2023-02-03 14:42:59,706 [MainThread] INFO UNet3DTrainer - Training iteration [1711/60000]. Epoch [30/499]
2023-02-03 14:43:00,280 [MainThread] INFO UNet3DTrainer - Training iteration [1712/60000]. Epoch [30/499]
2023-02-03 14:43:00,930 [MainThread] INFO UNet3DTrainer - Training iteration [1713/60000]. Epoch [30/499]
2023-02-03 14:43:01,598 [MainThread] INFO UNet3DTrainer - Training iteration [1714/60000]. Epoch [30/499]
2023-02-03 14:43:43,461 [MainThread] INFO UNet3DTrainer - Training iteration [1715/60000]. Epoch [30/499]
2023-02-03 14:43:44,659 [MainThread] INFO UNet3DTrainer - Training iteration [1716/60000]. Epoch [30/499]
2023-02-03 14:43:45,016 [MainThread] INFO UNet3DTrainer - Training iteration [1717/60000]. Epoch [30/499]
2023-02-03 14:43:45,558 [MainThread] INFO UNet3DTrainer - Training iteration [1718/60000]. Epoch [30/499]
2023-02-03 14:44:08,698 [MainThread] INFO UNet3DTrainer - Training iteration [1719/60000]. Epoch [30/499]
2023-02-03 14:44:09,089 [MainThread] INFO UNet3DTrainer - Training iteration [1720/60000]. Epoch [30/499]
2023-02-03 14:44:09,654 [MainThread] INFO UNet3DTrainer - Training iteration [1721/60000]. Epoch [30/499]
2023-02-03 14:44:10,244 [MainThread] INFO UNet3DTrainer - Training iteration [1722/60000]. Epoch [30/499]
2023-02-03 14:44:10,820 [MainThread] INFO UNet3DTrainer - Training iteration [1723/60000]. Epoch [30/499]
2023-02-03 14:44:11,383 [MainThread] INFO UNet3DTrainer - Training iteration [1724/60000]. Epoch [30/499]
2023-02-03 14:44:34,256 [MainThread] INFO UNet3DTrainer - Training iteration [1725/60000]. Epoch [30/499]
2023-02-03 14:44:34,650 [MainThread] INFO UNet3DTrainer - Training iteration [1726/60000]. Epoch [30/499]
2023-02-03 14:44:36,506 [MainThread] INFO UNet3DTrainer - Training iteration [1727/60000]. Epoch [30/499]
2023-02-03 14:44:36,839 [MainThread] INFO UNet3DTrainer - Training iteration [1728/60000]. Epoch [30/499]
2023-02-03 14:44:37,376 [MainThread] INFO UNet3DTrainer - Training iteration [1729/60000]. Epoch [30/499]
2023-02-03 14:44:37,913 [MainThread] INFO UNet3DTrainer - Training iteration [1730/60000]. Epoch [30/499]
2023-02-03 14:45:13,927 [MainThread] INFO UNet3DTrainer - Training iteration [1731/60000]. Epoch [30/499]
2023-02-03 14:45:14,297 [MainThread] INFO UNet3DTrainer - Training iteration [1732/60000]. Epoch [30/499]
2023-02-03 14:45:14,820 [MainThread] INFO UNet3DTrainer - Training iteration [1733/60000]. Epoch [30/499]
2023-02-03 14:45:15,352 [MainThread] INFO UNet3DTrainer - Training iteration [1734/60000]. Epoch [30/499]
2023-02-03 14:45:15,882 [MainThread] INFO UNet3DTrainer - Training iteration [1735/60000]. Epoch [30/499]
2023-02-03 14:45:16,415 [MainThread] INFO UNet3DTrainer - Training iteration [1736/60000]. Epoch [30/499]
2023-02-03 14:45:43,427 [MainThread] INFO UNet3DTrainer - Training iteration [1737/60000]. Epoch [31/499]
2023-02-03 14:45:43,833 [MainThread] INFO UNet3DTrainer - Training iteration [1738/60000]. Epoch [31/499]
2023-02-03 14:45:44,386 [MainThread] INFO UNet3DTrainer - Training iteration [1739/60000]. Epoch [31/499]
2023-02-03 14:45:44,936 [MainThread] INFO UNet3DTrainer - Training iteration [1740/60000]. Epoch [31/499]
2023-02-03 14:45:45,539 [MainThread] INFO UNet3DTrainer - Training iteration [1741/60000]. Epoch [31/499]
2023-02-03 14:45:46,092 [MainThread] INFO UNet3DTrainer - Training iteration [1742/60000]. Epoch [31/499]
2023-02-03 14:46:27,451 [MainThread] INFO UNet3DTrainer - Training iteration [1743/60000]. Epoch [31/499]
2023-02-03 14:46:27,832 [MainThread] INFO UNet3DTrainer - Training iteration [1744/60000]. Epoch [31/499]
2023-02-03 14:46:28,367 [MainThread] INFO UNet3DTrainer - Training iteration [1745/60000]. Epoch [31/499]
2023-02-03 14:46:28,916 [MainThread] INFO UNet3DTrainer - Training iteration [1746/60000]. Epoch [31/499]
2023-02-03 14:46:29,463 [MainThread] INFO UNet3DTrainer - Training iteration [1747/60000]. Epoch [31/499]
2023-02-03 14:46:30,070 [MainThread] INFO UNet3DTrainer - Training iteration [1748/60000]. Epoch [31/499]
2023-02-03 14:46:32,398 [MainThread] INFO UNet3DTrainer - Training iteration [1749/60000]. Epoch [31/499]
2023-02-03 14:46:32,790 [MainThread] INFO UNet3DTrainer - Training iteration [1750/60000]. Epoch [31/499]
2023-02-03 14:46:33,410 [MainThread] INFO UNet3DTrainer - Training iteration [1751/60000]. Epoch [31/499]
2023-02-03 14:46:54,160 [MainThread] INFO UNet3DTrainer - Training iteration [1752/60000]. Epoch [31/499]
2023-02-03 14:46:54,547 [MainThread] INFO UNet3DTrainer - Training iteration [1753/60000]. Epoch [31/499]
2023-02-03 14:46:55,090 [MainThread] INFO UNet3DTrainer - Training iteration [1754/60000]. Epoch [31/499]
2023-02-03 14:46:55,636 [MainThread] INFO UNet3DTrainer - Training iteration [1755/60000]. Epoch [31/499]
2023-02-03 14:46:56,233 [MainThread] INFO UNet3DTrainer - Training iteration [1756/60000]. Epoch [31/499]
2023-02-03 14:46:56,830 [MainThread] INFO UNet3DTrainer - Training iteration [1757/60000]. Epoch [31/499]
2023-02-03 14:47:17,585 [MainThread] INFO UNet3DTrainer - Training iteration [1758/60000]. Epoch [31/499]
2023-02-03 14:47:17,972 [MainThread] INFO UNet3DTrainer - Training iteration [1759/60000]. Epoch [31/499]
2023-02-03 14:47:18,525 [MainThread] INFO UNet3DTrainer - Training iteration [1760/60000]. Epoch [31/499]
2023-02-03 14:47:19,066 [MainThread] INFO UNet3DTrainer - Training iteration [1761/60000]. Epoch [31/499]
2023-02-03 14:47:19,607 [MainThread] INFO UNet3DTrainer - Training iteration [1762/60000]. Epoch [31/499]
2023-02-03 14:47:20,169 [MainThread] INFO UNet3DTrainer - Training iteration [1763/60000]. Epoch [31/499]
2023-02-03 14:47:22,771 [MainThread] INFO UNet3DTrainer - Training iteration [1764/60000]. Epoch [31/499]
2023-02-03 14:47:23,260 [MainThread] INFO UNet3DTrainer - Training iteration [1765/60000]. Epoch [31/499]
2023-02-03 14:47:43,821 [MainThread] INFO UNet3DTrainer - Training iteration [1766/60000]. Epoch [31/499]
2023-02-03 14:47:44,215 [MainThread] INFO UNet3DTrainer - Training iteration [1767/60000]. Epoch [31/499]
2023-02-03 14:47:44,941 [MainThread] INFO UNet3DTrainer - Training iteration [1768/60000]. Epoch [31/499]
2023-02-03 14:47:45,513 [MainThread] INFO UNet3DTrainer - Training iteration [1769/60000]. Epoch [31/499]
2023-02-03 14:47:50,632 [MainThread] INFO UNet3DTrainer - Training iteration [1770/60000]. Epoch [31/499]
2023-02-03 14:48:11,113 [MainThread] INFO UNet3DTrainer - Training iteration [1771/60000]. Epoch [31/499]
2023-02-03 14:48:11,507 [MainThread] INFO UNet3DTrainer - Training iteration [1772/60000]. Epoch [31/499]
2023-02-03 14:48:12,057 [MainThread] INFO UNet3DTrainer - Training iteration [1773/60000]. Epoch [31/499]
2023-02-03 14:48:12,750 [MainThread] INFO UNet3DTrainer - Training iteration [1774/60000]. Epoch [31/499]
2023-02-03 14:48:36,688 [MainThread] INFO UNet3DTrainer - Training iteration [1775/60000]. Epoch [31/499]
2023-02-03 14:48:37,078 [MainThread] INFO UNet3DTrainer - Training iteration [1776/60000]. Epoch [31/499]
2023-02-03 14:48:37,633 [MainThread] INFO UNet3DTrainer - Training iteration [1777/60000]. Epoch [31/499]
2023-02-03 14:48:38,190 [MainThread] INFO UNet3DTrainer - Training iteration [1778/60000]. Epoch [31/499]
2023-02-03 14:48:38,794 [MainThread] INFO UNet3DTrainer - Training iteration [1779/60000]. Epoch [31/499]
2023-02-03 14:48:39,397 [MainThread] INFO UNet3DTrainer - Training iteration [1780/60000]. Epoch [31/499]
2023-02-03 14:49:00,602 [MainThread] INFO UNet3DTrainer - Training iteration [1781/60000]. Epoch [31/499]
2023-02-03 14:49:00,976 [MainThread] INFO UNet3DTrainer - Training iteration [1782/60000]. Epoch [31/499]
2023-02-03 14:49:01,519 [MainThread] INFO UNet3DTrainer - Training iteration [1783/60000]. Epoch [31/499]
2023-02-03 14:49:02,052 [MainThread] INFO UNet3DTrainer - Training iteration [1784/60000]. Epoch [31/499]
2023-02-03 14:49:02,589 [MainThread] INFO UNet3DTrainer - Training iteration [1785/60000]. Epoch [31/499]
2023-02-03 14:49:03,110 [MainThread] INFO UNet3DTrainer - Training iteration [1786/60000]. Epoch [31/499]
2023-02-03 14:49:03,649 [MainThread] INFO UNet3DTrainer - Training iteration [1787/60000]. Epoch [31/499]
2023-02-03 14:49:04,189 [MainThread] INFO UNet3DTrainer - Training iteration [1788/60000]. Epoch [31/499]
2023-02-03 14:49:05,000 [MainThread] INFO UNet3DTrainer - Training iteration [1789/60000]. Epoch [31/499]
2023-02-03 14:49:05,337 [MainThread] INFO UNet3DTrainer - Training iteration [1790/60000]. Epoch [31/499]
2023-02-03 14:49:05,854 [MainThread] INFO UNet3DTrainer - Training iteration [1791/60000]. Epoch [31/499]
2023-02-03 14:49:06,386 [MainThread] INFO UNet3DTrainer - Training iteration [1792/60000]. Epoch [31/499]
2023-02-03 14:49:35,983 [MainThread] INFO UNet3DTrainer - Training iteration [1793/60000]. Epoch [32/499]
2023-02-03 14:49:36,380 [MainThread] INFO UNet3DTrainer - Training iteration [1794/60000]. Epoch [32/499]
2023-02-03 14:49:37,694 [MainThread] INFO UNet3DTrainer - Training iteration [1795/60000]. Epoch [32/499]
2023-02-03 14:49:57,364 [MainThread] INFO UNet3DTrainer - Training iteration [1796/60000]. Epoch [32/499]
2023-02-03 14:49:57,756 [MainThread] INFO UNet3DTrainer - Training iteration [1797/60000]. Epoch [32/499]
2023-02-03 14:49:58,313 [MainThread] INFO UNet3DTrainer - Training iteration [1798/60000]. Epoch [32/499]
2023-02-03 14:49:58,872 [MainThread] INFO UNet3DTrainer - Training iteration [1799/60000]. Epoch [32/499]
2023-02-03 14:49:59,419 [MainThread] INFO UNet3DTrainer - Training iteration [1800/60000]. Epoch [32/499]
2023-02-03 14:49:59,951 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 14:50:06,831 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 14:50:19,436 [MainThread] INFO EvalMetric - ARand: 0.6487568983866091
2023-02-03 14:50:19,478 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 14:50:28,217 [MainThread] INFO EvalMetric - ARand: 0.6747400885222911
2023-02-03 14:50:28,283 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 14:50:36,323 [MainThread] INFO EvalMetric - ARand: 0.7203633114541216
2023-02-03 14:50:36,362 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 14:50:43,750 [MainThread] INFO EvalMetric - ARand: 0.6680002413722594
2023-02-03 14:50:43,792 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 14:50:51,254 [MainThread] INFO EvalMetric - ARand: 0.6469316550297125
2023-02-03 14:50:51,292 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 14:50:58,725 [MainThread] INFO EvalMetric - ARand: 0.728942787242949
2023-02-03 14:50:58,771 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 14:51:06,313 [MainThread] INFO EvalMetric - ARand: 0.665211344469041
2023-02-03 14:51:06,355 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 14:51:13,759 [MainThread] INFO EvalMetric - ARand: 0.7173218079140063
2023-02-03 14:51:13,797 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 14:51:21,207 [MainThread] INFO EvalMetric - ARand: 0.6558459207475695
2023-02-03 14:51:21,248 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 14:51:28,629 [MainThread] INFO EvalMetric - ARand: 0.6388630022625422
2023-02-03 14:51:28,669 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 14:51:36,088 [MainThread] INFO EvalMetric - ARand: 0.5990052944645665
2023-02-03 14:51:36,124 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 14:51:43,457 [MainThread] INFO EvalMetric - ARand: 0.6384211540076474
2023-02-03 14:51:43,500 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 14:51:50,944 [MainThread] INFO EvalMetric - ARand: 0.6206335429553467
2023-02-03 14:51:50,982 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 14:51:58,520 [MainThread] INFO EvalMetric - ARand: 0.6985127753191467
2023-02-03 14:51:58,560 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 14:52:05,947 [MainThread] INFO EvalMetric - ARand: 0.6912964568264508
2023-02-03 14:52:05,993 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 14:52:13,487 [MainThread] INFO EvalMetric - ARand: 0.7074893982166304
2023-02-03 14:52:13,528 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 14:52:21,036 [MainThread] INFO EvalMetric - ARand: 0.6204434884914787
2023-02-03 14:52:21,078 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 14:52:28,527 [MainThread] INFO EvalMetric - ARand: 0.7608342401150954
2023-02-03 14:52:28,574 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 14:52:36,130 [MainThread] INFO EvalMetric - ARand: 0.6815315037946161
2023-02-03 14:52:36,176 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 14:52:43,721 [MainThread] INFO EvalMetric - ARand: 0.6834235312519733
2023-02-03 14:52:43,766 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 14:52:51,274 [MainThread] INFO EvalMetric - ARand: 0.7370775637529152
2023-02-03 14:52:51,318 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 14:52:57,004 [MainThread] INFO EvalMetric - ARand: 0.7126373872731432
2023-02-03 14:52:57,395 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4763733545939128. Evaluation score: 0.6776148987150264
2023-02-03 14:52:57,412 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 14:53:04,679 [MainThread] INFO EvalMetric - ARand: 0.7634855845521942
2023-02-03 14:53:04,699 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.15398839861154556. Evaluation score: 0.7634855845521942
2023-02-03 14:53:04,700 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 14:53:05,704 [MainThread] INFO UNet3DTrainer - Training iteration [1801/60000]. Epoch [32/499]
2023-02-03 14:53:06,068 [MainThread] INFO UNet3DTrainer - Training iteration [1802/60000]. Epoch [32/499]
2023-02-03 14:53:06,598 [MainThread] INFO UNet3DTrainer - Training iteration [1803/60000]. Epoch [32/499]
2023-02-03 14:53:07,162 [MainThread] INFO UNet3DTrainer - Training iteration [1804/60000]. Epoch [32/499]
2023-02-03 14:53:07,706 [MainThread] INFO UNet3DTrainer - Training iteration [1805/60000]. Epoch [32/499]
2023-02-03 14:53:08,249 [MainThread] INFO UNet3DTrainer - Training iteration [1806/60000]. Epoch [32/499]
2023-02-03 14:53:08,866 [MainThread] INFO UNet3DTrainer - Training iteration [1807/60000]. Epoch [32/499]
2023-02-03 14:53:09,520 [MainThread] INFO UNet3DTrainer - Training iteration [1808/60000]. Epoch [32/499]
2023-02-03 14:53:10,091 [MainThread] INFO UNet3DTrainer - Training iteration [1809/60000]. Epoch [32/499]
2023-02-03 14:53:10,660 [MainThread] INFO UNet3DTrainer - Training iteration [1810/60000]. Epoch [32/499]
2023-02-03 14:53:11,188 [MainThread] INFO UNet3DTrainer - Training iteration [1811/60000]. Epoch [32/499]
2023-02-03 14:53:11,800 [MainThread] INFO UNet3DTrainer - Training iteration [1812/60000]. Epoch [32/499]
2023-02-03 14:53:12,441 [MainThread] INFO UNet3DTrainer - Training iteration [1813/60000]. Epoch [32/499]
2023-02-03 14:53:31,270 [MainThread] INFO UNet3DTrainer - Training iteration [1814/60000]. Epoch [32/499]
2023-02-03 14:53:49,818 [MainThread] INFO UNet3DTrainer - Training iteration [1815/60000]. Epoch [32/499]
2023-02-03 14:53:50,198 [MainThread] INFO UNet3DTrainer - Training iteration [1816/60000]. Epoch [32/499]
2023-02-03 14:53:50,747 [MainThread] INFO UNet3DTrainer - Training iteration [1817/60000]. Epoch [32/499]
2023-02-03 14:53:51,305 [MainThread] INFO UNet3DTrainer - Training iteration [1818/60000]. Epoch [32/499]
2023-02-03 14:53:51,834 [MainThread] INFO UNet3DTrainer - Training iteration [1819/60000]. Epoch [32/499]
2023-02-03 14:53:52,407 [MainThread] INFO UNet3DTrainer - Training iteration [1820/60000]. Epoch [32/499]
2023-02-03 14:53:54,153 [MainThread] INFO UNet3DTrainer - Training iteration [1821/60000]. Epoch [32/499]
2023-02-03 14:53:54,551 [MainThread] INFO UNet3DTrainer - Training iteration [1822/60000]. Epoch [32/499]
2023-02-03 14:53:55,252 [MainThread] INFO UNet3DTrainer - Training iteration [1823/60000]. Epoch [32/499]
2023-02-03 14:53:55,815 [MainThread] INFO UNet3DTrainer - Training iteration [1824/60000]. Epoch [32/499]
2023-02-03 14:53:56,470 [MainThread] INFO UNet3DTrainer - Training iteration [1825/60000]. Epoch [32/499]
2023-02-03 14:53:57,048 [MainThread] INFO UNet3DTrainer - Training iteration [1826/60000]. Epoch [32/499]
2023-02-03 14:53:59,883 [MainThread] INFO UNet3DTrainer - Training iteration [1827/60000]. Epoch [32/499]
2023-02-03 14:54:00,287 [MainThread] INFO UNet3DTrainer - Training iteration [1828/60000]. Epoch [32/499]
2023-02-03 14:54:23,911 [MainThread] INFO UNet3DTrainer - Training iteration [1829/60000]. Epoch [32/499]
2023-02-03 14:54:24,307 [MainThread] INFO UNet3DTrainer - Training iteration [1830/60000]. Epoch [32/499]
2023-02-03 14:54:24,877 [MainThread] INFO UNet3DTrainer - Training iteration [1831/60000]. Epoch [32/499]
2023-02-03 14:54:25,432 [MainThread] INFO UNet3DTrainer - Training iteration [1832/60000]. Epoch [32/499]
2023-02-03 14:54:26,010 [MainThread] INFO UNet3DTrainer - Training iteration [1833/60000]. Epoch [32/499]
2023-02-03 14:54:26,547 [MainThread] INFO UNet3DTrainer - Training iteration [1834/60000]. Epoch [32/499]
2023-02-03 14:55:30,784 [MainThread] INFO UNet3DTrainer - Training iteration [1835/60000]. Epoch [32/499]
2023-02-03 14:55:31,157 [MainThread] INFO UNet3DTrainer - Training iteration [1836/60000]. Epoch [32/499]
2023-02-03 14:55:31,695 [MainThread] INFO UNet3DTrainer - Training iteration [1837/60000]. Epoch [32/499]
2023-02-03 14:55:32,226 [MainThread] INFO UNet3DTrainer - Training iteration [1838/60000]. Epoch [32/499]
2023-02-03 14:55:32,753 [MainThread] INFO UNet3DTrainer - Training iteration [1839/60000]. Epoch [32/499]
2023-02-03 14:55:33,296 [MainThread] INFO UNet3DTrainer - Training iteration [1840/60000]. Epoch [32/499]
2023-02-03 14:55:34,101 [MainThread] INFO UNet3DTrainer - Training iteration [1841/60000]. Epoch [32/499]
2023-02-03 14:55:34,436 [MainThread] INFO UNet3DTrainer - Training iteration [1842/60000]. Epoch [32/499]
2023-02-03 14:55:34,963 [MainThread] INFO UNet3DTrainer - Training iteration [1843/60000]. Epoch [32/499]
2023-02-03 14:55:35,498 [MainThread] INFO UNet3DTrainer - Training iteration [1844/60000]. Epoch [32/499]
2023-02-03 14:55:36,025 [MainThread] INFO UNet3DTrainer - Training iteration [1845/60000]. Epoch [32/499]
2023-02-03 14:55:36,552 [MainThread] INFO UNet3DTrainer - Training iteration [1846/60000]. Epoch [32/499]
2023-02-03 14:55:37,090 [MainThread] INFO UNet3DTrainer - Training iteration [1847/60000]. Epoch [32/499]
2023-02-03 14:55:51,618 [MainThread] INFO UNet3DTrainer - Training iteration [1848/60000]. Epoch [32/499]
2023-02-03 14:55:59,731 [MainThread] INFO UNet3DTrainer - Training iteration [1849/60000]. Epoch [33/499]
2023-02-03 14:56:23,276 [MainThread] INFO UNet3DTrainer - Training iteration [1850/60000]. Epoch [33/499]
2023-02-03 14:56:23,671 [MainThread] INFO UNet3DTrainer - Training iteration [1851/60000]. Epoch [33/499]
2023-02-03 14:56:24,228 [MainThread] INFO UNet3DTrainer - Training iteration [1852/60000]. Epoch [33/499]
2023-02-03 14:56:24,838 [MainThread] INFO UNet3DTrainer - Training iteration [1853/60000]. Epoch [33/499]
2023-02-03 14:56:25,370 [MainThread] INFO UNet3DTrainer - Training iteration [1854/60000]. Epoch [33/499]
2023-02-03 14:56:28,360 [MainThread] INFO UNet3DTrainer - Training iteration [1855/60000]. Epoch [33/499]
2023-02-03 14:56:50,090 [MainThread] INFO UNet3DTrainer - Training iteration [1856/60000]. Epoch [33/499]
2023-02-03 14:56:50,471 [MainThread] INFO UNet3DTrainer - Training iteration [1857/60000]. Epoch [33/499]
2023-02-03 14:56:51,025 [MainThread] INFO UNet3DTrainer - Training iteration [1858/60000]. Epoch [33/499]
2023-02-03 14:56:51,569 [MainThread] INFO UNet3DTrainer - Training iteration [1859/60000]. Epoch [33/499]
2023-02-03 14:57:10,222 [MainThread] INFO UNet3DTrainer - Training iteration [1860/60000]. Epoch [33/499]
2023-02-03 14:57:10,600 [MainThread] INFO UNet3DTrainer - Training iteration [1861/60000]. Epoch [33/499]
2023-02-03 14:57:13,376 [MainThread] INFO UNet3DTrainer - Training iteration [1862/60000]. Epoch [33/499]
2023-02-03 14:57:13,762 [MainThread] INFO UNet3DTrainer - Training iteration [1863/60000]. Epoch [33/499]
2023-02-03 14:57:14,329 [MainThread] INFO UNet3DTrainer - Training iteration [1864/60000]. Epoch [33/499]
2023-02-03 14:57:14,910 [MainThread] INFO UNet3DTrainer - Training iteration [1865/60000]. Epoch [33/499]
2023-02-03 14:57:37,036 [MainThread] INFO UNet3DTrainer - Training iteration [1866/60000]. Epoch [33/499]
2023-02-03 14:57:37,434 [MainThread] INFO UNet3DTrainer - Training iteration [1867/60000]. Epoch [33/499]
2023-02-03 14:57:37,982 [MainThread] INFO UNet3DTrainer - Training iteration [1868/60000]. Epoch [33/499]
2023-02-03 14:57:38,536 [MainThread] INFO UNet3DTrainer - Training iteration [1869/60000]. Epoch [33/499]
2023-02-03 14:57:39,090 [MainThread] INFO UNet3DTrainer - Training iteration [1870/60000]. Epoch [33/499]
2023-02-03 14:57:39,756 [MainThread] INFO UNet3DTrainer - Training iteration [1871/60000]. Epoch [33/499]
2023-02-03 14:57:42,550 [MainThread] INFO UNet3DTrainer - Training iteration [1872/60000]. Epoch [33/499]
2023-02-03 14:57:43,021 [MainThread] INFO UNet3DTrainer - Training iteration [1873/60000]. Epoch [33/499]
2023-02-03 14:57:43,600 [MainThread] INFO UNet3DTrainer - Training iteration [1874/60000]. Epoch [33/499]
2023-02-03 14:57:44,192 [MainThread] INFO UNet3DTrainer - Training iteration [1875/60000]. Epoch [33/499]
2023-02-03 14:57:44,740 [MainThread] INFO UNet3DTrainer - Training iteration [1876/60000]. Epoch [33/499]
2023-02-03 14:58:08,012 [MainThread] INFO UNet3DTrainer - Training iteration [1877/60000]. Epoch [33/499]
2023-02-03 14:58:13,508 [MainThread] INFO UNet3DTrainer - Training iteration [1878/60000]. Epoch [33/499]
2023-02-03 14:58:13,892 [MainThread] INFO UNet3DTrainer - Training iteration [1879/60000]. Epoch [33/499]
2023-02-03 14:58:14,486 [MainThread] INFO UNet3DTrainer - Training iteration [1880/60000]. Epoch [33/499]
2023-02-03 14:58:35,723 [MainThread] INFO UNet3DTrainer - Training iteration [1881/60000]. Epoch [33/499]
2023-02-03 14:58:36,126 [MainThread] INFO UNet3DTrainer - Training iteration [1882/60000]. Epoch [33/499]
2023-02-03 14:58:36,693 [MainThread] INFO UNet3DTrainer - Training iteration [1883/60000]. Epoch [33/499]
2023-02-03 14:58:37,242 [MainThread] INFO UNet3DTrainer - Training iteration [1884/60000]. Epoch [33/499]
2023-02-03 14:58:37,792 [MainThread] INFO UNet3DTrainer - Training iteration [1885/60000]. Epoch [33/499]
2023-02-03 14:58:38,392 [MainThread] INFO UNet3DTrainer - Training iteration [1886/60000]. Epoch [33/499]
2023-02-03 14:59:20,613 [MainThread] INFO UNet3DTrainer - Training iteration [1887/60000]. Epoch [33/499]
2023-02-03 14:59:20,988 [MainThread] INFO UNet3DTrainer - Training iteration [1888/60000]. Epoch [33/499]
2023-02-03 14:59:21,517 [MainThread] INFO UNet3DTrainer - Training iteration [1889/60000]. Epoch [33/499]
2023-02-03 14:59:22,050 [MainThread] INFO UNet3DTrainer - Training iteration [1890/60000]. Epoch [33/499]
2023-02-03 14:59:22,604 [MainThread] INFO UNet3DTrainer - Training iteration [1891/60000]. Epoch [33/499]
2023-02-03 14:59:23,167 [MainThread] INFO UNet3DTrainer - Training iteration [1892/60000]. Epoch [33/499]
2023-02-03 14:59:24,683 [MainThread] INFO UNet3DTrainer - Training iteration [1893/60000]. Epoch [33/499]
2023-02-03 14:59:25,174 [MainThread] INFO UNet3DTrainer - Training iteration [1894/60000]. Epoch [33/499]
2023-02-03 14:59:25,774 [MainThread] INFO UNet3DTrainer - Training iteration [1895/60000]. Epoch [33/499]
2023-02-03 14:59:26,296 [MainThread] INFO UNet3DTrainer - Training iteration [1896/60000]. Epoch [33/499]
2023-02-03 14:59:26,853 [MainThread] INFO UNet3DTrainer - Training iteration [1897/60000]. Epoch [33/499]
2023-02-03 14:59:27,380 [MainThread] INFO UNet3DTrainer - Training iteration [1898/60000]. Epoch [33/499]
2023-02-03 14:59:28,524 [MainThread] INFO UNet3DTrainer - Training iteration [1899/60000]. Epoch [33/499]
2023-02-03 14:59:28,854 [MainThread] INFO UNet3DTrainer - Training iteration [1900/60000]. Epoch [33/499]
2023-02-03 14:59:29,363 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 14:59:33,701 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 14:59:42,688 [MainThread] INFO EvalMetric - ARand: 0.6740396830588985
2023-02-03 14:59:42,730 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 14:59:50,053 [MainThread] INFO EvalMetric - ARand: 0.7812477271708227
2023-02-03 14:59:50,093 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 14:59:57,653 [MainThread] INFO EvalMetric - ARand: 0.7536176789252573
2023-02-03 14:59:57,693 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 15:00:05,072 [MainThread] INFO EvalMetric - ARand: 0.744631345552748
2023-02-03 15:00:05,111 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 15:00:12,613 [MainThread] INFO EvalMetric - ARand: 0.742348879085847
2023-02-03 15:00:12,653 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 15:00:19,961 [MainThread] INFO EvalMetric - ARand: 0.7595177376167804
2023-02-03 15:00:20,007 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 15:00:27,380 [MainThread] INFO EvalMetric - ARand: 0.6909144388868698
2023-02-03 15:00:27,418 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 15:00:34,732 [MainThread] INFO EvalMetric - ARand: 0.7360796950243675
2023-02-03 15:00:34,776 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 15:00:42,189 [MainThread] INFO EvalMetric - ARand: 0.7074434141571386
2023-02-03 15:00:42,227 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 15:00:49,655 [MainThread] INFO EvalMetric - ARand: 0.6809606848251439
2023-02-03 15:00:49,690 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 15:00:57,047 [MainThread] INFO EvalMetric - ARand: 0.5961148597128062
2023-02-03 15:00:57,086 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 15:01:04,465 [MainThread] INFO EvalMetric - ARand: 0.6760771393576008
2023-02-03 15:01:04,507 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 15:01:11,898 [MainThread] INFO EvalMetric - ARand: 0.6716048728739819
2023-02-03 15:01:11,943 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 15:01:19,363 [MainThread] INFO EvalMetric - ARand: 0.714716641147732
2023-02-03 15:01:19,410 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 15:01:26,818 [MainThread] INFO EvalMetric - ARand: 0.7344603924471234
2023-02-03 15:01:26,862 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 15:01:34,321 [MainThread] INFO EvalMetric - ARand: 0.7444518994791185
2023-02-03 15:01:34,365 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 15:01:41,886 [MainThread] INFO EvalMetric - ARand: 0.6732889636123025
2023-02-03 15:01:41,929 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 15:01:49,375 [MainThread] INFO EvalMetric - ARand: 0.7700758237630566
2023-02-03 15:01:49,419 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 15:01:56,955 [MainThread] INFO EvalMetric - ARand: 0.7368099575262326
2023-02-03 15:01:57,000 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 15:02:04,550 [MainThread] INFO EvalMetric - ARand: 0.698971171248092
2023-02-03 15:02:04,596 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 15:02:12,128 [MainThread] INFO EvalMetric - ARand: 0.773806423587575
2023-02-03 15:02:12,168 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 15:02:17,850 [MainThread] INFO EvalMetric - ARand: 0.7665110311471807
2023-02-03 15:02:18,240 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.477525800123982. Evaluation score: 0.7188994345940174
2023-02-03 15:02:18,262 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 15:02:25,590 [MainThread] INFO EvalMetric - ARand: 0.771300631017444
2023-02-03 15:02:25,611 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.16034984273406175. Evaluation score: 0.771300631017444
2023-02-03 15:02:25,612 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 15:02:26,705 [MainThread] INFO UNet3DTrainer - Training iteration [1901/60000]. Epoch [33/499]
2023-02-03 15:02:27,085 [MainThread] INFO UNet3DTrainer - Training iteration [1902/60000]. Epoch [33/499]
2023-02-03 15:02:27,614 [MainThread] INFO UNet3DTrainer - Training iteration [1903/60000]. Epoch [33/499]
2023-02-03 15:02:28,142 [MainThread] INFO UNet3DTrainer - Training iteration [1904/60000]. Epoch [33/499]
2023-02-03 15:02:55,064 [MainThread] INFO UNet3DTrainer - Training iteration [1905/60000]. Epoch [34/499]
2023-02-03 15:02:55,446 [MainThread] INFO UNet3DTrainer - Training iteration [1906/60000]. Epoch [34/499]
2023-02-03 15:02:56,000 [MainThread] INFO UNet3DTrainer - Training iteration [1907/60000]. Epoch [34/499]
2023-02-03 15:02:56,538 [MainThread] INFO UNet3DTrainer - Training iteration [1908/60000]. Epoch [34/499]
2023-02-03 15:02:57,093 [MainThread] INFO UNet3DTrainer - Training iteration [1909/60000]. Epoch [34/499]
2023-02-03 15:02:57,655 [MainThread] INFO UNet3DTrainer - Training iteration [1910/60000]. Epoch [34/499]
2023-02-03 15:03:59,413 [MainThread] INFO UNet3DTrainer - Training iteration [1911/60000]. Epoch [34/499]
2023-02-03 15:03:59,790 [MainThread] INFO UNet3DTrainer - Training iteration [1912/60000]. Epoch [34/499]
2023-02-03 15:04:00,319 [MainThread] INFO UNet3DTrainer - Training iteration [1913/60000]. Epoch [34/499]
2023-02-03 15:04:00,870 [MainThread] INFO UNet3DTrainer - Training iteration [1914/60000]. Epoch [34/499]
2023-02-03 15:04:01,412 [MainThread] INFO UNet3DTrainer - Training iteration [1915/60000]. Epoch [34/499]
2023-02-03 15:04:01,989 [MainThread] INFO UNet3DTrainer - Training iteration [1916/60000]. Epoch [34/499]
2023-02-03 15:04:39,961 [MainThread] INFO UNet3DTrainer - Training iteration [1917/60000]. Epoch [34/499]
2023-02-03 15:04:40,347 [MainThread] INFO UNet3DTrainer - Training iteration [1918/60000]. Epoch [34/499]
2023-02-03 15:04:40,898 [MainThread] INFO UNet3DTrainer - Training iteration [1919/60000]. Epoch [34/499]
2023-02-03 15:04:41,453 [MainThread] INFO UNet3DTrainer - Training iteration [1920/60000]. Epoch [34/499]
2023-02-03 15:04:42,012 [MainThread] INFO UNet3DTrainer - Training iteration [1921/60000]. Epoch [34/499]
2023-02-03 15:04:42,560 [MainThread] INFO UNet3DTrainer - Training iteration [1922/60000]. Epoch [34/499]
2023-02-03 15:04:44,240 [MainThread] INFO UNet3DTrainer - Training iteration [1923/60000]. Epoch [34/499]
2023-02-03 15:04:44,703 [MainThread] INFO UNet3DTrainer - Training iteration [1924/60000]. Epoch [34/499]
2023-02-03 15:04:45,290 [MainThread] INFO UNet3DTrainer - Training iteration [1925/60000]. Epoch [34/499]
2023-02-03 15:04:45,870 [MainThread] INFO UNet3DTrainer - Training iteration [1926/60000]. Epoch [34/499]
2023-02-03 15:04:46,470 [MainThread] INFO UNet3DTrainer - Training iteration [1927/60000]. Epoch [34/499]
2023-02-03 15:04:47,030 [MainThread] INFO UNet3DTrainer - Training iteration [1928/60000]. Epoch [34/499]
2023-02-03 15:05:12,941 [MainThread] INFO UNet3DTrainer - Training iteration [1929/60000]. Epoch [34/499]
2023-02-03 15:05:13,353 [MainThread] INFO UNet3DTrainer - Training iteration [1930/60000]. Epoch [34/499]
2023-02-03 15:05:13,903 [MainThread] INFO UNet3DTrainer - Training iteration [1931/60000]. Epoch [34/499]
2023-02-03 15:05:34,840 [MainThread] INFO UNet3DTrainer - Training iteration [1932/60000]. Epoch [34/499]
2023-02-03 15:05:35,221 [MainThread] INFO UNet3DTrainer - Training iteration [1933/60000]. Epoch [34/499]
2023-02-03 15:05:35,773 [MainThread] INFO UNet3DTrainer - Training iteration [1934/60000]. Epoch [34/499]
2023-02-03 15:05:36,321 [MainThread] INFO UNet3DTrainer - Training iteration [1935/60000]. Epoch [34/499]
2023-02-03 15:05:36,907 [MainThread] INFO UNet3DTrainer - Training iteration [1936/60000]. Epoch [34/499]
2023-02-03 15:05:37,485 [MainThread] INFO UNet3DTrainer - Training iteration [1937/60000]. Epoch [34/499]
2023-02-03 15:06:03,359 [MainThread] INFO UNet3DTrainer - Training iteration [1938/60000]. Epoch [34/499]
2023-02-03 15:06:03,764 [MainThread] INFO UNet3DTrainer - Training iteration [1939/60000]. Epoch [34/499]
2023-02-03 15:06:04,335 [MainThread] INFO UNet3DTrainer - Training iteration [1940/60000]. Epoch [34/499]
2023-02-03 15:06:04,893 [MainThread] INFO UNet3DTrainer - Training iteration [1941/60000]. Epoch [34/499]
2023-02-03 15:06:05,446 [MainThread] INFO UNet3DTrainer - Training iteration [1942/60000]. Epoch [34/499]
2023-02-03 15:06:06,016 [MainThread] INFO UNet3DTrainer - Training iteration [1943/60000]. Epoch [34/499]
2023-02-03 15:06:08,499 [MainThread] INFO UNet3DTrainer - Training iteration [1944/60000]. Epoch [34/499]
2023-02-03 15:06:08,925 [MainThread] INFO UNet3DTrainer - Training iteration [1945/60000]. Epoch [34/499]
2023-02-03 15:06:09,470 [MainThread] INFO UNet3DTrainer - Training iteration [1946/60000]. Epoch [34/499]
2023-02-03 15:06:26,177 [MainThread] INFO UNet3DTrainer - Training iteration [1947/60000]. Epoch [34/499]
2023-02-03 15:06:28,231 [MainThread] INFO UNet3DTrainer - Training iteration [1948/60000]. Epoch [34/499]
2023-02-03 15:06:28,572 [MainThread] INFO UNet3DTrainer - Training iteration [1949/60000]. Epoch [34/499]
2023-02-03 15:06:29,093 [MainThread] INFO UNet3DTrainer - Training iteration [1950/60000]. Epoch [34/499]
2023-02-03 15:06:29,626 [MainThread] INFO UNet3DTrainer - Training iteration [1951/60000]. Epoch [34/499]
2023-02-03 15:06:30,154 [MainThread] INFO UNet3DTrainer - Training iteration [1952/60000]. Epoch [34/499]
2023-02-03 15:06:30,700 [MainThread] INFO UNet3DTrainer - Training iteration [1953/60000]. Epoch [34/499]
2023-02-03 15:06:31,238 [MainThread] INFO UNet3DTrainer - Training iteration [1954/60000]. Epoch [34/499]
2023-02-03 15:06:31,764 [MainThread] INFO UNet3DTrainer - Training iteration [1955/60000]. Epoch [34/499]
2023-02-03 15:06:32,290 [MainThread] INFO UNet3DTrainer - Training iteration [1956/60000]. Epoch [34/499]
2023-02-03 15:06:32,826 [MainThread] INFO UNet3DTrainer - Training iteration [1957/60000]. Epoch [34/499]
2023-02-03 15:06:33,352 [MainThread] INFO UNet3DTrainer - Training iteration [1958/60000]. Epoch [34/499]
2023-02-03 15:06:50,069 [MainThread] INFO UNet3DTrainer - Training iteration [1959/60000]. Epoch [34/499]
2023-02-03 15:06:50,447 [MainThread] INFO UNet3DTrainer - Training iteration [1960/60000]. Epoch [34/499]
2023-02-03 15:06:58,255 [MainThread] INFO UNet3DTrainer - Training iteration [1961/60000]. Epoch [35/499]
2023-02-03 15:06:58,778 [MainThread] INFO UNet3DTrainer - Training iteration [1962/60000]. Epoch [35/499]
2023-02-03 15:07:20,111 [MainThread] INFO UNet3DTrainer - Training iteration [1963/60000]. Epoch [35/499]
2023-02-03 15:07:20,699 [MainThread] INFO UNet3DTrainer - Training iteration [1964/60000]. Epoch [35/499]
2023-02-03 15:07:21,078 [MainThread] INFO UNet3DTrainer - Training iteration [1965/60000]. Epoch [35/499]
2023-02-03 15:07:43,531 [MainThread] INFO UNet3DTrainer - Training iteration [1966/60000]. Epoch [35/499]
2023-02-03 15:07:43,914 [MainThread] INFO UNet3DTrainer - Training iteration [1967/60000]. Epoch [35/499]
2023-02-03 15:07:44,499 [MainThread] INFO UNet3DTrainer - Training iteration [1968/60000]. Epoch [35/499]
2023-02-03 15:07:45,030 [MainThread] INFO UNet3DTrainer - Training iteration [1969/60000]. Epoch [35/499]
2023-02-03 15:07:45,646 [MainThread] INFO UNet3DTrainer - Training iteration [1970/60000]. Epoch [35/499]
2023-02-03 15:07:46,276 [MainThread] INFO UNet3DTrainer - Training iteration [1971/60000]. Epoch [35/499]
2023-02-03 15:07:48,922 [MainThread] INFO UNet3DTrainer - Training iteration [1972/60000]. Epoch [35/499]
2023-02-03 15:07:49,390 [MainThread] INFO UNet3DTrainer - Training iteration [1973/60000]. Epoch [35/499]
2023-02-03 15:07:49,970 [MainThread] INFO UNet3DTrainer - Training iteration [1974/60000]. Epoch [35/499]
2023-02-03 15:07:51,335 [MainThread] INFO UNet3DTrainer - Training iteration [1975/60000]. Epoch [35/499]
2023-02-03 15:07:51,762 [MainThread] INFO UNet3DTrainer - Training iteration [1976/60000]. Epoch [35/499]
2023-02-03 15:07:52,430 [MainThread] INFO UNet3DTrainer - Training iteration [1977/60000]. Epoch [35/499]
2023-02-03 15:07:54,246 [MainThread] INFO UNet3DTrainer - Training iteration [1978/60000]. Epoch [35/499]
2023-02-03 15:07:54,756 [MainThread] INFO UNet3DTrainer - Training iteration [1979/60000]. Epoch [35/499]
2023-02-03 15:08:15,570 [MainThread] INFO UNet3DTrainer - Training iteration [1980/60000]. Epoch [35/499]
2023-02-03 15:08:21,764 [MainThread] INFO UNet3DTrainer - Training iteration [1981/60000]. Epoch [35/499]
2023-02-03 15:08:22,520 [MainThread] INFO UNet3DTrainer - Training iteration [1982/60000]. Epoch [35/499]
2023-02-03 15:08:23,012 [MainThread] INFO UNet3DTrainer - Training iteration [1983/60000]. Epoch [35/499]
2023-02-03 15:08:24,942 [MainThread] INFO UNet3DTrainer - Training iteration [1984/60000]. Epoch [35/499]
2023-02-03 15:08:25,321 [MainThread] INFO UNet3DTrainer - Training iteration [1985/60000]. Epoch [35/499]
2023-02-03 15:08:25,867 [MainThread] INFO UNet3DTrainer - Training iteration [1986/60000]. Epoch [35/499]
2023-02-03 15:08:26,450 [MainThread] INFO UNet3DTrainer - Training iteration [1987/60000]. Epoch [35/499]
2023-02-03 15:08:27,410 [MainThread] INFO UNet3DTrainer - Training iteration [1988/60000]. Epoch [35/499]
2023-02-03 15:08:27,830 [MainThread] INFO UNet3DTrainer - Training iteration [1989/60000]. Epoch [35/499]
2023-02-03 15:08:54,008 [MainThread] INFO UNet3DTrainer - Training iteration [1990/60000]. Epoch [35/499]
2023-02-03 15:08:54,418 [MainThread] INFO UNet3DTrainer - Training iteration [1991/60000]. Epoch [35/499]
2023-02-03 15:08:54,978 [MainThread] INFO UNet3DTrainer - Training iteration [1992/60000]. Epoch [35/499]
2023-02-03 15:08:55,665 [MainThread] INFO UNet3DTrainer - Training iteration [1993/60000]. Epoch [35/499]
2023-02-03 15:08:56,311 [MainThread] INFO UNet3DTrainer - Training iteration [1994/60000]. Epoch [35/499]
2023-02-03 15:08:56,880 [MainThread] INFO UNet3DTrainer - Training iteration [1995/60000]. Epoch [35/499]
2023-02-03 15:08:59,049 [MainThread] INFO UNet3DTrainer - Training iteration [1996/60000]. Epoch [35/499]
2023-02-03 15:09:18,630 [MainThread] INFO UNet3DTrainer - Training iteration [1997/60000]. Epoch [35/499]
2023-02-03 15:09:19,070 [MainThread] INFO UNet3DTrainer - Training iteration [1998/60000]. Epoch [35/499]
2023-02-03 15:09:25,070 [MainThread] INFO UNet3DTrainer - Training iteration [1999/60000]. Epoch [35/499]
2023-02-03 15:09:25,516 [MainThread] INFO UNet3DTrainer - Training iteration [2000/60000]. Epoch [35/499]
2023-02-03 15:09:26,056 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 15:09:32,272 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 15:09:43,324 [MainThread] INFO EvalMetric - ARand: 0.6674609403414311
2023-02-03 15:09:43,366 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 15:09:52,565 [MainThread] INFO EvalMetric - ARand: 0.677899749486255
2023-02-03 15:09:52,623 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 15:10:02,118 [MainThread] INFO EvalMetric - ARand: 0.7021339780195801
2023-02-03 15:10:02,166 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 15:10:09,791 [MainThread] INFO EvalMetric - ARand: 0.6485874265490925
2023-02-03 15:10:09,841 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 15:10:17,533 [MainThread] INFO EvalMetric - ARand: 0.6515035587974697
2023-02-03 15:10:17,574 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 15:10:25,215 [MainThread] INFO EvalMetric - ARand: 0.7040395319064633
2023-02-03 15:10:25,260 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 15:10:32,789 [MainThread] INFO EvalMetric - ARand: 0.6280788112566131
2023-02-03 15:10:32,832 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 15:10:40,509 [MainThread] INFO EvalMetric - ARand: 0.6979772498017173
2023-02-03 15:10:40,549 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 15:10:47,955 [MainThread] INFO EvalMetric - ARand: 0.6584909963586019
2023-02-03 15:10:47,995 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 15:10:55,500 [MainThread] INFO EvalMetric - ARand: 0.6119872751344835
2023-02-03 15:10:55,538 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 15:11:03,040 [MainThread] INFO EvalMetric - ARand: 0.5985107422245233
2023-02-03 15:11:03,084 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 15:11:10,430 [MainThread] INFO EvalMetric - ARand: 0.6240189761153246
2023-02-03 15:11:10,471 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 15:11:17,877 [MainThread] INFO EvalMetric - ARand: 0.6195014679951705
2023-02-03 15:11:17,918 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 15:11:25,310 [MainThread] INFO EvalMetric - ARand: 0.6905694100400348
2023-02-03 15:11:25,352 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 15:11:32,801 [MainThread] INFO EvalMetric - ARand: 0.6589794926483628
2023-02-03 15:11:32,847 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 15:11:40,290 [MainThread] INFO EvalMetric - ARand: 0.6841766582810954
2023-02-03 15:11:40,332 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 15:11:47,831 [MainThread] INFO EvalMetric - ARand: 0.6342422262822526
2023-02-03 15:11:47,870 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 15:11:55,374 [MainThread] INFO EvalMetric - ARand: 0.7209728180754005
2023-02-03 15:11:55,423 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 15:12:03,033 [MainThread] INFO EvalMetric - ARand: 0.6951580986877036
2023-02-03 15:12:03,073 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 15:12:10,559 [MainThread] INFO EvalMetric - ARand: 0.677149360934341
2023-02-03 15:12:10,605 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 15:12:18,104 [MainThread] INFO EvalMetric - ARand: 0.7227981773784409
2023-02-03 15:12:18,156 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 15:12:23,839 [MainThread] INFO EvalMetric - ARand: 0.6934713365724537
2023-02-03 15:12:24,235 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4769545437275677. Evaluation score: 0.6664064574135035
2023-02-03 15:12:24,264 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 15:12:31,588 [MainThread] INFO EvalMetric - ARand: 0.7981414287950022
2023-02-03 15:12:31,609 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.16073668878525496. Evaluation score: 0.7981414287950022
2023-02-03 15:12:31,610 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 15:12:32,696 [MainThread] INFO UNet3DTrainer - Training iteration [2001/60000]. Epoch [35/499]
2023-02-03 15:12:33,067 [MainThread] INFO UNet3DTrainer - Training iteration [2002/60000]. Epoch [35/499]
2023-02-03 15:12:33,607 [MainThread] INFO UNet3DTrainer - Training iteration [2003/60000]. Epoch [35/499]
2023-02-03 15:12:34,153 [MainThread] INFO UNet3DTrainer - Training iteration [2004/60000]. Epoch [35/499]
2023-02-03 15:12:34,718 [MainThread] INFO UNet3DTrainer - Training iteration [2005/60000]. Epoch [35/499]
2023-02-03 15:12:35,249 [MainThread] INFO UNet3DTrainer - Training iteration [2006/60000]. Epoch [35/499]
2023-02-03 15:12:35,780 [MainThread] INFO UNet3DTrainer - Training iteration [2007/60000]. Epoch [35/499]
2023-02-03 15:12:36,316 [MainThread] INFO UNet3DTrainer - Training iteration [2008/60000]. Epoch [35/499]
2023-02-03 15:12:36,858 [MainThread] INFO UNet3DTrainer - Training iteration [2009/60000]. Epoch [35/499]
2023-02-03 15:12:37,402 [MainThread] INFO UNet3DTrainer - Training iteration [2010/60000]. Epoch [35/499]
2023-02-03 15:12:37,919 [MainThread] INFO UNet3DTrainer - Training iteration [2011/60000]. Epoch [35/499]
2023-02-03 15:12:38,451 [MainThread] INFO UNet3DTrainer - Training iteration [2012/60000]. Epoch [35/499]
2023-02-03 15:12:38,987 [MainThread] INFO UNet3DTrainer - Training iteration [2013/60000]. Epoch [35/499]
2023-02-03 15:12:39,518 [MainThread] INFO UNet3DTrainer - Training iteration [2014/60000]. Epoch [35/499]
2023-02-03 15:12:40,046 [MainThread] INFO UNet3DTrainer - Training iteration [2015/60000]. Epoch [35/499]
2023-02-03 15:12:40,586 [MainThread] INFO UNet3DTrainer - Training iteration [2016/60000]. Epoch [35/499]
2023-02-03 15:12:48,230 [MainThread] INFO UNet3DTrainer - Training iteration [2017/60000]. Epoch [36/499]
2023-02-03 15:13:12,215 [MainThread] INFO UNet3DTrainer - Training iteration [2018/60000]. Epoch [36/499]
2023-02-03 15:13:12,615 [MainThread] INFO UNet3DTrainer - Training iteration [2019/60000]. Epoch [36/499]
2023-02-03 15:13:13,216 [MainThread] INFO UNet3DTrainer - Training iteration [2020/60000]. Epoch [36/499]
2023-02-03 15:13:36,282 [MainThread] INFO UNet3DTrainer - Training iteration [2021/60000]. Epoch [36/499]
2023-02-03 15:13:36,676 [MainThread] INFO UNet3DTrainer - Training iteration [2022/60000]. Epoch [36/499]
2023-02-03 15:13:37,251 [MainThread] INFO UNet3DTrainer - Training iteration [2023/60000]. Epoch [36/499]
2023-02-03 15:13:42,511 [MainThread] INFO UNet3DTrainer - Training iteration [2024/60000]. Epoch [36/499]
2023-02-03 15:13:42,919 [MainThread] INFO UNet3DTrainer - Training iteration [2025/60000]. Epoch [36/499]
2023-02-03 15:13:43,460 [MainThread] INFO UNet3DTrainer - Training iteration [2026/60000]. Epoch [36/499]
2023-02-03 15:14:04,976 [MainThread] INFO UNet3DTrainer - Training iteration [2027/60000]. Epoch [36/499]
2023-02-03 15:14:06,251 [MainThread] INFO UNet3DTrainer - Training iteration [2028/60000]. Epoch [36/499]
2023-02-03 15:14:06,606 [MainThread] INFO UNet3DTrainer - Training iteration [2029/60000]. Epoch [36/499]
2023-02-03 15:14:10,636 [MainThread] INFO UNet3DTrainer - Training iteration [2030/60000]. Epoch [36/499]
2023-02-03 15:14:11,057 [MainThread] INFO UNet3DTrainer - Training iteration [2031/60000]. Epoch [36/499]
2023-02-03 15:14:11,650 [MainThread] INFO UNet3DTrainer - Training iteration [2032/60000]. Epoch [36/499]
2023-02-03 15:14:12,270 [MainThread] INFO UNet3DTrainer - Training iteration [2033/60000]. Epoch [36/499]
2023-02-03 15:14:12,840 [MainThread] INFO UNet3DTrainer - Training iteration [2034/60000]. Epoch [36/499]
2023-02-03 15:14:28,807 [MainThread] INFO UNet3DTrainer - Training iteration [2035/60000]. Epoch [36/499]
2023-02-03 15:14:29,196 [MainThread] INFO UNet3DTrainer - Training iteration [2036/60000]. Epoch [36/499]
2023-02-03 15:14:29,735 [MainThread] INFO UNet3DTrainer - Training iteration [2037/60000]. Epoch [36/499]
2023-02-03 15:14:34,894 [MainThread] INFO UNet3DTrainer - Training iteration [2038/60000]. Epoch [36/499]
2023-02-03 15:14:35,285 [MainThread] INFO UNet3DTrainer - Training iteration [2039/60000]. Epoch [36/499]
2023-02-03 15:14:35,825 [MainThread] INFO UNet3DTrainer - Training iteration [2040/60000]. Epoch [36/499]
2023-02-03 15:14:53,054 [MainThread] INFO UNet3DTrainer - Training iteration [2041/60000]. Epoch [36/499]
2023-02-03 15:14:53,439 [MainThread] INFO UNet3DTrainer - Training iteration [2042/60000]. Epoch [36/499]
2023-02-03 15:14:53,980 [MainThread] INFO UNet3DTrainer - Training iteration [2043/60000]. Epoch [36/499]
2023-02-03 15:14:54,529 [MainThread] INFO UNet3DTrainer - Training iteration [2044/60000]. Epoch [36/499]
2023-02-03 15:14:55,089 [MainThread] INFO UNet3DTrainer - Training iteration [2045/60000]. Epoch [36/499]
2023-02-03 15:14:55,669 [MainThread] INFO UNet3DTrainer - Training iteration [2046/60000]. Epoch [36/499]
2023-02-03 15:14:57,126 [MainThread] INFO UNet3DTrainer - Training iteration [2047/60000]. Epoch [36/499]
2023-02-03 15:14:57,540 [MainThread] INFO UNet3DTrainer - Training iteration [2048/60000]. Epoch [36/499]
2023-02-03 15:14:58,072 [MainThread] INFO UNet3DTrainer - Training iteration [2049/60000]. Epoch [36/499]
2023-02-03 15:14:58,639 [MainThread] INFO UNet3DTrainer - Training iteration [2050/60000]. Epoch [36/499]
2023-02-03 15:15:04,918 [MainThread] INFO UNet3DTrainer - Training iteration [2051/60000]. Epoch [36/499]
2023-02-03 15:15:05,318 [MainThread] INFO UNet3DTrainer - Training iteration [2052/60000]. Epoch [36/499]
2023-02-03 15:15:05,872 [MainThread] INFO UNet3DTrainer - Training iteration [2053/60000]. Epoch [36/499]
2023-02-03 15:15:19,337 [MainThread] INFO UNet3DTrainer - Training iteration [2054/60000]. Epoch [36/499]
2023-02-03 15:15:19,713 [MainThread] INFO UNet3DTrainer - Training iteration [2055/60000]. Epoch [36/499]
2023-02-03 15:15:20,265 [MainThread] INFO UNet3DTrainer - Training iteration [2056/60000]. Epoch [36/499]
2023-02-03 15:15:29,028 [MainThread] INFO UNet3DTrainer - Training iteration [2057/60000]. Epoch [36/499]
2023-02-03 15:15:29,403 [MainThread] INFO UNet3DTrainer - Training iteration [2058/60000]. Epoch [36/499]
2023-02-03 15:15:29,950 [MainThread] INFO UNet3DTrainer - Training iteration [2059/60000]. Epoch [36/499]
2023-02-03 15:16:20,683 [MainThread] INFO UNet3DTrainer - Training iteration [2060/60000]. Epoch [36/499]
2023-02-03 15:16:21,060 [MainThread] INFO UNet3DTrainer - Training iteration [2061/60000]. Epoch [36/499]
2023-02-03 15:16:21,591 [MainThread] INFO UNet3DTrainer - Training iteration [2062/60000]. Epoch [36/499]
2023-02-03 15:16:22,120 [MainThread] INFO UNet3DTrainer - Training iteration [2063/60000]. Epoch [36/499]
2023-02-03 15:16:22,648 [MainThread] INFO UNet3DTrainer - Training iteration [2064/60000]. Epoch [36/499]
2023-02-03 15:16:23,183 [MainThread] INFO UNet3DTrainer - Training iteration [2065/60000]. Epoch [36/499]
2023-02-03 15:16:41,816 [MainThread] INFO UNet3DTrainer - Training iteration [2066/60000]. Epoch [36/499]
2023-02-03 15:16:42,195 [MainThread] INFO UNet3DTrainer - Training iteration [2067/60000]. Epoch [36/499]
2023-02-03 15:16:42,726 [MainThread] INFO UNet3DTrainer - Training iteration [2068/60000]. Epoch [36/499]
2023-02-03 15:16:43,250 [MainThread] INFO UNet3DTrainer - Training iteration [2069/60000]. Epoch [36/499]
2023-02-03 15:16:43,784 [MainThread] INFO UNet3DTrainer - Training iteration [2070/60000]. Epoch [36/499]
2023-02-03 15:16:44,313 [MainThread] INFO UNet3DTrainer - Training iteration [2071/60000]. Epoch [36/499]
2023-02-03 15:16:44,846 [MainThread] INFO UNet3DTrainer - Training iteration [2072/60000]. Epoch [36/499]
2023-02-03 15:17:11,453 [MainThread] INFO UNet3DTrainer - Training iteration [2073/60000]. Epoch [37/499]
2023-02-03 15:17:11,848 [MainThread] INFO UNet3DTrainer - Training iteration [2074/60000]. Epoch [37/499]
2023-02-03 15:17:12,381 [MainThread] INFO UNet3DTrainer - Training iteration [2075/60000]. Epoch [37/499]
2023-02-03 15:17:12,934 [MainThread] INFO UNet3DTrainer - Training iteration [2076/60000]. Epoch [37/499]
2023-02-03 15:17:13,477 [MainThread] INFO UNet3DTrainer - Training iteration [2077/60000]. Epoch [37/499]
2023-02-03 15:17:14,031 [MainThread] INFO UNet3DTrainer - Training iteration [2078/60000]. Epoch [37/499]
2023-02-03 15:17:41,565 [MainThread] INFO UNet3DTrainer - Training iteration [2079/60000]. Epoch [37/499]
2023-02-03 15:17:41,983 [MainThread] INFO UNet3DTrainer - Training iteration [2080/60000]. Epoch [37/499]
2023-02-03 15:17:42,630 [MainThread] INFO UNet3DTrainer - Training iteration [2081/60000]. Epoch [37/499]
2023-02-03 15:17:43,170 [MainThread] INFO UNet3DTrainer - Training iteration [2082/60000]. Epoch [37/499]
2023-02-03 15:17:43,780 [MainThread] INFO UNet3DTrainer - Training iteration [2083/60000]. Epoch [37/499]
2023-02-03 15:17:44,419 [MainThread] INFO UNet3DTrainer - Training iteration [2084/60000]. Epoch [37/499]
2023-02-03 15:17:46,287 [MainThread] INFO UNet3DTrainer - Training iteration [2085/60000]. Epoch [37/499]
2023-02-03 15:17:46,715 [MainThread] INFO UNet3DTrainer - Training iteration [2086/60000]. Epoch [37/499]
2023-02-03 15:18:47,190 [MainThread] INFO UNet3DTrainer - Training iteration [2087/60000]. Epoch [37/499]
2023-02-03 15:18:47,573 [MainThread] INFO UNet3DTrainer - Training iteration [2088/60000]. Epoch [37/499]
2023-02-03 15:18:48,108 [MainThread] INFO UNet3DTrainer - Training iteration [2089/60000]. Epoch [37/499]
2023-02-03 15:18:48,642 [MainThread] INFO UNet3DTrainer - Training iteration [2090/60000]. Epoch [37/499]
2023-02-03 15:18:49,199 [MainThread] INFO UNet3DTrainer - Training iteration [2091/60000]. Epoch [37/499]
2023-02-03 15:18:49,741 [MainThread] INFO UNet3DTrainer - Training iteration [2092/60000]. Epoch [37/499]
2023-02-03 15:19:09,817 [MainThread] INFO UNet3DTrainer - Training iteration [2093/60000]. Epoch [37/499]
2023-02-03 15:19:10,200 [MainThread] INFO UNet3DTrainer - Training iteration [2094/60000]. Epoch [37/499]
2023-02-03 15:19:10,747 [MainThread] INFO UNet3DTrainer - Training iteration [2095/60000]. Epoch [37/499]
2023-02-03 15:19:11,296 [MainThread] INFO UNet3DTrainer - Training iteration [2096/60000]. Epoch [37/499]
2023-02-03 15:19:11,854 [MainThread] INFO UNet3DTrainer - Training iteration [2097/60000]. Epoch [37/499]
2023-02-03 15:19:12,420 [MainThread] INFO UNet3DTrainer - Training iteration [2098/60000]. Epoch [37/499]
2023-02-03 15:19:14,850 [MainThread] INFO UNet3DTrainer - Training iteration [2099/60000]. Epoch [37/499]
2023-02-03 15:19:15,316 [MainThread] INFO UNet3DTrainer - Training iteration [2100/60000]. Epoch [37/499]
2023-02-03 15:19:15,907 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 15:19:21,690 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 15:19:32,539 [MainThread] INFO EvalMetric - ARand: 0.6460643907827477
2023-02-03 15:19:32,581 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 15:19:41,111 [MainThread] INFO EvalMetric - ARand: 0.7019788179780349
2023-02-03 15:19:41,168 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 15:19:49,099 [MainThread] INFO EvalMetric - ARand: 0.7308202462790488
2023-02-03 15:19:49,146 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 15:19:56,689 [MainThread] INFO EvalMetric - ARand: 0.6708851235420263
2023-02-03 15:19:56,731 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 15:20:04,190 [MainThread] INFO EvalMetric - ARand: 0.6650894446574713
2023-02-03 15:20:04,235 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 15:20:11,647 [MainThread] INFO EvalMetric - ARand: 0.7302804719206968
2023-02-03 15:20:11,690 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 15:20:19,203 [MainThread] INFO EvalMetric - ARand: 0.6831241859575017
2023-02-03 15:20:19,243 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 15:20:26,632 [MainThread] INFO EvalMetric - ARand: 0.7168408614458939
2023-02-03 15:20:26,678 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 15:20:34,038 [MainThread] INFO EvalMetric - ARand: 0.64786490126445
2023-02-03 15:20:34,079 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 15:20:41,492 [MainThread] INFO EvalMetric - ARand: 0.6574254186702173
2023-02-03 15:20:41,532 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 15:20:48,987 [MainThread] INFO EvalMetric - ARand: 0.5720788131498642
2023-02-03 15:20:49,026 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 15:20:56,448 [MainThread] INFO EvalMetric - ARand: 0.6380340764352116
2023-02-03 15:20:56,491 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 15:21:03,872 [MainThread] INFO EvalMetric - ARand: 0.6375670393536819
2023-02-03 15:21:03,917 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 15:21:11,350 [MainThread] INFO EvalMetric - ARand: 0.7502092426307663
2023-02-03 15:21:11,392 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 15:21:18,831 [MainThread] INFO EvalMetric - ARand: 0.7333347333119846
2023-02-03 15:21:18,876 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 15:21:26,372 [MainThread] INFO EvalMetric - ARand: 0.7494321025826273
2023-02-03 15:21:26,413 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 15:21:33,930 [MainThread] INFO EvalMetric - ARand: 0.6405976680313179
2023-02-03 15:21:33,974 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 15:21:41,390 [MainThread] INFO EvalMetric - ARand: 0.7499629474283942
2023-02-03 15:21:41,436 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 15:21:48,858 [MainThread] INFO EvalMetric - ARand: 0.7473917684861586
2023-02-03 15:21:48,904 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 15:21:56,403 [MainThread] INFO EvalMetric - ARand: 0.7189760981690988
2023-02-03 15:21:56,452 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 15:22:03,971 [MainThread] INFO EvalMetric - ARand: 0.7680002578818073
2023-02-03 15:22:04,015 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 15:22:09,708 [MainThread] INFO EvalMetric - ARand: 0.7573670541553542
2023-02-03 15:22:10,097 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.476094463775898. Evaluation score: 0.6953555816356559
2023-02-03 15:22:10,123 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 15:22:17,504 [MainThread] INFO EvalMetric - ARand: 0.7226864018071548
2023-02-03 15:22:17,524 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.15242711268365383. Evaluation score: 0.7226864018071548
2023-02-03 15:22:17,525 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 15:22:18,586 [MainThread] INFO UNet3DTrainer - Training iteration [2101/60000]. Epoch [37/499]
2023-02-03 15:22:18,979 [MainThread] INFO UNet3DTrainer - Training iteration [2102/60000]. Epoch [37/499]
2023-02-03 15:22:19,511 [MainThread] INFO UNet3DTrainer - Training iteration [2103/60000]. Epoch [37/499]
2023-02-03 15:22:20,062 [MainThread] INFO UNet3DTrainer - Training iteration [2104/60000]. Epoch [37/499]
2023-02-03 15:22:20,586 [MainThread] INFO UNet3DTrainer - Training iteration [2105/60000]. Epoch [37/499]
2023-02-03 15:22:21,137 [MainThread] INFO UNet3DTrainer - Training iteration [2106/60000]. Epoch [37/499]
2023-02-03 15:22:21,802 [MainThread] INFO UNet3DTrainer - Training iteration [2107/60000]. Epoch [37/499]
2023-02-03 15:22:22,376 [MainThread] INFO UNet3DTrainer - Training iteration [2108/60000]. Epoch [37/499]
2023-02-03 15:22:22,927 [MainThread] INFO UNet3DTrainer - Training iteration [2109/60000]. Epoch [37/499]
2023-02-03 15:22:23,530 [MainThread] INFO UNet3DTrainer - Training iteration [2110/60000]. Epoch [37/499]
2023-02-03 15:22:24,100 [MainThread] INFO UNet3DTrainer - Training iteration [2111/60000]. Epoch [37/499]
2023-02-03 15:22:24,740 [MainThread] INFO UNet3DTrainer - Training iteration [2112/60000]. Epoch [37/499]
2023-02-03 15:23:06,324 [MainThread] INFO UNet3DTrainer - Training iteration [2113/60000]. Epoch [37/499]
2023-02-03 15:23:07,810 [MainThread] INFO UNet3DTrainer - Training iteration [2114/60000]. Epoch [37/499]
2023-02-03 15:23:08,163 [MainThread] INFO UNet3DTrainer - Training iteration [2115/60000]. Epoch [37/499]
2023-02-03 15:23:08,693 [MainThread] INFO UNet3DTrainer - Training iteration [2116/60000]. Epoch [37/499]
2023-02-03 15:23:09,236 [MainThread] INFO UNet3DTrainer - Training iteration [2117/60000]. Epoch [37/499]
2023-02-03 15:23:09,783 [MainThread] INFO UNet3DTrainer - Training iteration [2118/60000]. Epoch [37/499]
2023-02-03 15:23:10,339 [MainThread] INFO UNet3DTrainer - Training iteration [2119/60000]. Epoch [37/499]
2023-02-03 15:23:29,984 [MainThread] INFO UNet3DTrainer - Training iteration [2120/60000]. Epoch [37/499]
2023-02-03 15:23:30,359 [MainThread] INFO UNet3DTrainer - Training iteration [2121/60000]. Epoch [37/499]
2023-02-03 15:23:30,898 [MainThread] INFO UNet3DTrainer - Training iteration [2122/60000]. Epoch [37/499]
2023-02-03 15:23:31,434 [MainThread] INFO UNet3DTrainer - Training iteration [2123/60000]. Epoch [37/499]
2023-02-03 15:23:31,961 [MainThread] INFO UNet3DTrainer - Training iteration [2124/60000]. Epoch [37/499]
2023-02-03 15:23:32,498 [MainThread] INFO UNet3DTrainer - Training iteration [2125/60000]. Epoch [37/499]
2023-02-03 15:23:33,334 [MainThread] INFO UNet3DTrainer - Training iteration [2126/60000]. Epoch [37/499]
2023-02-03 15:23:33,666 [MainThread] INFO UNet3DTrainer - Training iteration [2127/60000]. Epoch [37/499]
2023-02-03 15:23:34,192 [MainThread] INFO UNet3DTrainer - Training iteration [2128/60000]. Epoch [37/499]
2023-02-03 15:23:41,979 [MainThread] INFO UNet3DTrainer - Training iteration [2129/60000]. Epoch [38/499]
2023-02-03 15:23:42,500 [MainThread] INFO UNet3DTrainer - Training iteration [2130/60000]. Epoch [38/499]
2023-02-03 15:24:05,967 [MainThread] INFO UNet3DTrainer - Training iteration [2131/60000]. Epoch [38/499]
2023-02-03 15:24:06,748 [MainThread] INFO UNet3DTrainer - Training iteration [2132/60000]. Epoch [38/499]
2023-02-03 15:24:07,122 [MainThread] INFO UNet3DTrainer - Training iteration [2133/60000]. Epoch [38/499]
2023-02-03 15:24:07,686 [MainThread] INFO UNet3DTrainer - Training iteration [2134/60000]. Epoch [38/499]
2023-02-03 15:24:08,248 [MainThread] INFO UNet3DTrainer - Training iteration [2135/60000]. Epoch [38/499]
2023-02-03 15:24:11,220 [MainThread] INFO UNet3DTrainer - Training iteration [2136/60000]. Epoch [38/499]
2023-02-03 15:24:30,685 [MainThread] INFO UNet3DTrainer - Training iteration [2137/60000]. Epoch [38/499]
2023-02-03 15:24:48,834 [MainThread] INFO UNet3DTrainer - Training iteration [2138/60000]. Epoch [38/499]
2023-02-03 15:24:49,194 [MainThread] INFO UNet3DTrainer - Training iteration [2139/60000]. Epoch [38/499]
2023-02-03 15:24:49,732 [MainThread] INFO UNet3DTrainer - Training iteration [2140/60000]. Epoch [38/499]
2023-02-03 15:24:50,268 [MainThread] INFO UNet3DTrainer - Training iteration [2141/60000]. Epoch [38/499]
2023-02-03 15:24:50,831 [MainThread] INFO UNet3DTrainer - Training iteration [2142/60000]. Epoch [38/499]
2023-02-03 15:24:53,271 [MainThread] INFO UNet3DTrainer - Training iteration [2143/60000]. Epoch [38/499]
2023-02-03 15:24:53,803 [MainThread] INFO UNet3DTrainer - Training iteration [2144/60000]. Epoch [38/499]
2023-02-03 15:24:54,378 [MainThread] INFO UNet3DTrainer - Training iteration [2145/60000]. Epoch [38/499]
2023-02-03 15:24:54,914 [MainThread] INFO UNet3DTrainer - Training iteration [2146/60000]. Epoch [38/499]
2023-02-03 15:24:55,472 [MainThread] INFO UNet3DTrainer - Training iteration [2147/60000]. Epoch [38/499]
2023-02-03 15:24:56,017 [MainThread] INFO UNet3DTrainer - Training iteration [2148/60000]. Epoch [38/499]
2023-02-03 15:25:37,761 [MainThread] INFO UNet3DTrainer - Training iteration [2149/60000]. Epoch [38/499]
2023-02-03 15:25:38,137 [MainThread] INFO UNet3DTrainer - Training iteration [2150/60000]. Epoch [38/499]
2023-02-03 15:25:38,674 [MainThread] INFO UNet3DTrainer - Training iteration [2151/60000]. Epoch [38/499]
2023-02-03 15:25:39,212 [MainThread] INFO UNet3DTrainer - Training iteration [2152/60000]. Epoch [38/499]
2023-02-03 15:25:39,774 [MainThread] INFO UNet3DTrainer - Training iteration [2153/60000]. Epoch [38/499]
2023-02-03 15:25:40,326 [MainThread] INFO UNet3DTrainer - Training iteration [2154/60000]. Epoch [38/499]
2023-02-03 15:26:19,003 [MainThread] INFO UNet3DTrainer - Training iteration [2155/60000]. Epoch [38/499]
2023-02-03 15:26:19,371 [MainThread] INFO UNet3DTrainer - Training iteration [2156/60000]. Epoch [38/499]
2023-02-03 15:26:19,895 [MainThread] INFO UNet3DTrainer - Training iteration [2157/60000]. Epoch [38/499]
2023-02-03 15:26:20,449 [MainThread] INFO UNet3DTrainer - Training iteration [2158/60000]. Epoch [38/499]
2023-02-03 15:26:21,000 [MainThread] INFO UNet3DTrainer - Training iteration [2159/60000]. Epoch [38/499]
2023-02-03 15:26:21,543 [MainThread] INFO UNet3DTrainer - Training iteration [2160/60000]. Epoch [38/499]
2023-02-03 15:26:41,227 [MainThread] INFO UNet3DTrainer - Training iteration [2161/60000]. Epoch [38/499]
2023-02-03 15:26:41,615 [MainThread] INFO UNet3DTrainer - Training iteration [2162/60000]. Epoch [38/499]
2023-02-03 15:26:42,152 [MainThread] INFO UNet3DTrainer - Training iteration [2163/60000]. Epoch [38/499]
2023-02-03 15:26:42,710 [MainThread] INFO UNet3DTrainer - Training iteration [2164/60000]. Epoch [38/499]
2023-02-03 15:26:43,277 [MainThread] INFO UNet3DTrainer - Training iteration [2165/60000]. Epoch [38/499]
2023-02-03 15:26:43,870 [MainThread] INFO UNet3DTrainer - Training iteration [2166/60000]. Epoch [38/499]
2023-02-03 15:27:08,375 [MainThread] INFO UNet3DTrainer - Training iteration [2167/60000]. Epoch [38/499]
2023-02-03 15:27:08,787 [MainThread] INFO UNet3DTrainer - Training iteration [2168/60000]. Epoch [38/499]
2023-02-03 15:27:09,341 [MainThread] INFO UNet3DTrainer - Training iteration [2169/60000]. Epoch [38/499]
2023-02-03 15:27:09,899 [MainThread] INFO UNet3DTrainer - Training iteration [2170/60000]. Epoch [38/499]
2023-02-03 15:27:10,465 [MainThread] INFO UNet3DTrainer - Training iteration [2171/60000]. Epoch [38/499]
2023-02-03 15:27:11,039 [MainThread] INFO UNet3DTrainer - Training iteration [2172/60000]. Epoch [38/499]
2023-02-03 15:27:33,733 [MainThread] INFO UNet3DTrainer - Training iteration [2173/60000]. Epoch [38/499]
2023-02-03 15:29:28,012 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 15:29:37,138 [MainThread] INFO EvalMetric - ARand: 0.6325574009673898
2023-02-03 15:29:37,175 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 15:29:44,559 [MainThread] INFO EvalMetric - ARand: 0.6486315673973096
2023-02-03 15:29:44,602 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 15:29:51,963 [MainThread] INFO EvalMetric - ARand: 0.7085906802070401
2023-02-03 15:29:52,001 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 15:29:59,447 [MainThread] INFO EvalMetric - ARand: 0.6421536168181186
2023-02-03 15:29:59,487 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 15:30:06,905 [MainThread] INFO EvalMetric - ARand: 0.6299012797775041
2023-02-03 15:30:06,942 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 15:30:14,323 [MainThread] INFO EvalMetric - ARand: 0.6746798844420506
2023-02-03 15:30:14,368 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 15:30:21,858 [MainThread] INFO EvalMetric - ARand: 0.6831617436448152
2023-02-03 15:30:21,898 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 15:30:29,343 [MainThread] INFO EvalMetric - ARand: 0.7062525368029345
2023-02-03 15:30:29,377 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 15:30:36,682 [MainThread] INFO EvalMetric - ARand: 0.6925181298165624
2023-02-03 15:30:36,726 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 15:30:44,085 [MainThread] INFO EvalMetric - ARand: 0.5701388996584946
2023-02-03 15:30:44,124 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 15:30:51,581 [MainThread] INFO EvalMetric - ARand: 0.5635223697100745
2023-02-03 15:30:51,617 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 15:30:58,998 [MainThread] INFO EvalMetric - ARand: 0.5676593270917074
2023-02-03 15:30:59,037 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 15:31:06,410 [MainThread] INFO EvalMetric - ARand: 0.6072193075419758
2023-02-03 15:31:06,452 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 15:31:13,871 [MainThread] INFO EvalMetric - ARand: 0.6893555004538879
2023-02-03 15:31:13,914 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 15:31:21,372 [MainThread] INFO EvalMetric - ARand: 0.6922784758154157
2023-02-03 15:31:21,421 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 15:31:28,884 [MainThread] INFO EvalMetric - ARand: 0.6620354372849727
2023-02-03 15:31:28,926 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 15:31:36,469 [MainThread] INFO EvalMetric - ARand: 0.623299279092762
2023-02-03 15:31:36,513 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 15:31:44,060 [MainThread] INFO EvalMetric - ARand: 0.7219892023421147
2023-02-03 15:31:44,104 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 15:31:51,681 [MainThread] INFO EvalMetric - ARand: 0.6404898509632702
2023-02-03 15:31:51,725 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 15:31:59,320 [MainThread] INFO EvalMetric - ARand: 0.6467581066342163
2023-02-03 15:31:59,365 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 15:32:06,816 [MainThread] INFO EvalMetric - ARand: 0.6881859215488759
2023-02-03 15:32:06,858 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 15:32:12,563 [MainThread] INFO EvalMetric - ARand: 0.6711889413038458
2023-02-03 15:32:12,958 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4753040664497463. Evaluation score: 0.652633113746638
2023-02-03 15:32:12,984 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 15:32:20,299 [MainThread] INFO EvalMetric - ARand: 0.7420279342806916
2023-02-03 15:32:20,319 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.1468092119321227. Evaluation score: 0.7420279342806916
2023-02-03 15:32:20,320 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 15:32:21,369 [MainThread] INFO UNet3DTrainer - Training iteration [2201/60000]. Epoch [39/499]
2023-02-03 15:32:21,749 [MainThread] INFO UNet3DTrainer - Training iteration [2202/60000]. Epoch [39/499]
2023-02-03 15:32:22,297 [MainThread] INFO UNet3DTrainer - Training iteration [2203/60000]. Epoch [39/499]
2023-02-03 15:32:22,826 [MainThread] INFO UNet3DTrainer - Training iteration [2204/60000]. Epoch [39/499]
2023-02-03 15:32:23,372 [MainThread] INFO UNet3DTrainer - Training iteration [2205/60000]. Epoch [39/499]
2023-02-03 15:32:23,937 [MainThread] INFO UNet3DTrainer - Training iteration [2206/60000]. Epoch [39/499]
2023-02-03 15:32:24,540 [MainThread] INFO UNet3DTrainer - Training iteration [2207/60000]. Epoch [39/499]
2023-02-03 15:32:25,070 [MainThread] INFO UNet3DTrainer - Training iteration [2208/60000]. Epoch [39/499]
2023-02-03 15:32:25,698 [MainThread] INFO UNet3DTrainer - Training iteration [2209/60000]. Epoch [39/499]
2023-02-03 15:32:26,287 [MainThread] INFO UNet3DTrainer - Training iteration [2210/60000]. Epoch [39/499]
2023-02-03 15:32:26,831 [MainThread] INFO UNet3DTrainer - Training iteration [2211/60000]. Epoch [39/499]
2023-02-03 15:32:27,420 [MainThread] INFO UNet3DTrainer - Training iteration [2212/60000]. Epoch [39/499]
2023-02-03 15:32:28,070 [MainThread] INFO UNet3DTrainer - Training iteration [2213/60000]. Epoch [39/499]
2023-02-03 15:32:28,689 [MainThread] INFO UNet3DTrainer - Training iteration [2214/60000]. Epoch [39/499]
2023-02-03 15:32:51,020 [MainThread] INFO UNet3DTrainer - Training iteration [2215/60000]. Epoch [39/499]
2023-02-03 15:32:51,439 [MainThread] INFO UNet3DTrainer - Training iteration [2216/60000]. Epoch [39/499]
2023-02-03 15:33:16,903 [MainThread] INFO UNet3DTrainer - Training iteration [2217/60000]. Epoch [39/499]
2023-02-03 15:33:17,300 [MainThread] INFO UNet3DTrainer - Training iteration [2218/60000]. Epoch [39/499]
2023-02-03 15:33:17,883 [MainThread] INFO UNet3DTrainer - Training iteration [2219/60000]. Epoch [39/499]
2023-02-03 15:33:18,480 [MainThread] INFO UNet3DTrainer - Training iteration [2220/60000]. Epoch [39/499]
2023-02-03 15:33:19,053 [MainThread] INFO UNet3DTrainer - Training iteration [2221/60000]. Epoch [39/499]
2023-02-03 15:33:20,909 [MainThread] INFO UNet3DTrainer - Training iteration [2222/60000]. Epoch [39/499]
2023-02-03 15:33:21,650 [MainThread] INFO UNet3DTrainer - Training iteration [2223/60000]. Epoch [39/499]
2023-02-03 15:33:22,393 [MainThread] INFO UNet3DTrainer - Training iteration [2224/60000]. Epoch [39/499]
2023-02-03 15:33:22,810 [MainThread] INFO UNet3DTrainer - Training iteration [2225/60000]. Epoch [39/499]
2023-02-03 15:33:23,473 [MainThread] INFO UNet3DTrainer - Training iteration [2226/60000]. Epoch [39/499]
2023-02-03 15:33:24,080 [MainThread] INFO UNet3DTrainer - Training iteration [2227/60000]. Epoch [39/499]
2023-02-03 15:33:45,449 [MainThread] INFO UNet3DTrainer - Training iteration [2228/60000]. Epoch [39/499]
2023-02-03 15:33:46,531 [MainThread] INFO UNet3DTrainer - Training iteration [2229/60000]. Epoch [39/499]
2023-02-03 15:33:46,873 [MainThread] INFO UNet3DTrainer - Training iteration [2230/60000]. Epoch [39/499]
2023-02-03 15:34:02,818 [MainThread] INFO UNet3DTrainer - Training iteration [2231/60000]. Epoch [39/499]
2023-02-03 15:34:03,189 [MainThread] INFO UNet3DTrainer - Training iteration [2232/60000]. Epoch [39/499]
2023-02-03 15:34:03,726 [MainThread] INFO UNet3DTrainer - Training iteration [2233/60000]. Epoch [39/499]
2023-02-03 15:34:07,119 [MainThread] INFO UNet3DTrainer - Training iteration [2234/60000]. Epoch [39/499]
2023-02-03 15:34:26,427 [MainThread] INFO UNet3DTrainer - Training iteration [2235/60000]. Epoch [39/499]
2023-02-03 15:34:26,787 [MainThread] INFO UNet3DTrainer - Training iteration [2236/60000]. Epoch [39/499]
2023-02-03 15:34:27,320 [MainThread] INFO UNet3DTrainer - Training iteration [2237/60000]. Epoch [39/499]
2023-02-03 15:34:27,853 [MainThread] INFO UNet3DTrainer - Training iteration [2238/60000]. Epoch [39/499]
2023-02-03 15:34:28,385 [MainThread] INFO UNet3DTrainer - Training iteration [2239/60000]. Epoch [39/499]
2023-02-03 15:34:28,915 [MainThread] INFO UNet3DTrainer - Training iteration [2240/60000]. Epoch [39/499]
2023-02-03 15:34:59,844 [MainThread] INFO UNet3DTrainer - Training iteration [2241/60000]. Epoch [40/499]
2023-02-03 15:35:16,624 [MainThread] INFO UNet3DTrainer - Training iteration [2242/60000]. Epoch [40/499]
2023-02-03 15:35:16,996 [MainThread] INFO UNet3DTrainer - Training iteration [2243/60000]. Epoch [40/499]
2023-02-03 15:35:17,552 [MainThread] INFO UNet3DTrainer - Training iteration [2244/60000]. Epoch [40/499]
2023-02-03 15:35:18,106 [MainThread] INFO UNet3DTrainer - Training iteration [2245/60000]. Epoch [40/499]
2023-02-03 15:35:18,650 [MainThread] INFO UNet3DTrainer - Training iteration [2246/60000]. Epoch [40/499]
2023-02-03 15:35:19,205 [MainThread] INFO UNet3DTrainer - Training iteration [2247/60000]. Epoch [40/499]
2023-02-03 15:35:20,550 [MainThread] INFO UNet3DTrainer - Training iteration [2248/60000]. Epoch [40/499]
2023-02-03 15:35:20,937 [MainThread] INFO UNet3DTrainer - Training iteration [2249/60000]. Epoch [40/499]
2023-02-03 15:35:21,608 [MainThread] INFO UNet3DTrainer - Training iteration [2250/60000]. Epoch [40/499]
2023-02-03 15:35:40,907 [MainThread] INFO UNet3DTrainer - Training iteration [2251/60000]. Epoch [40/499]
2023-02-03 15:35:41,298 [MainThread] INFO UNet3DTrainer - Training iteration [2252/60000]. Epoch [40/499]
2023-02-03 15:35:41,851 [MainThread] INFO UNet3DTrainer - Training iteration [2253/60000]. Epoch [40/499]
2023-02-03 15:35:42,392 [MainThread] INFO UNet3DTrainer - Training iteration [2254/60000]. Epoch [40/499]
2023-02-03 15:36:18,630 [MainThread] INFO UNet3DTrainer - Training iteration [2255/60000]. Epoch [40/499]
2023-02-03 15:36:19,003 [MainThread] INFO UNet3DTrainer - Training iteration [2256/60000]. Epoch [40/499]
2023-02-03 15:36:19,553 [MainThread] INFO UNet3DTrainer - Training iteration [2257/60000]. Epoch [40/499]
2023-02-03 15:36:20,081 [MainThread] INFO UNet3DTrainer - Training iteration [2258/60000]. Epoch [40/499]
2023-02-03 15:36:20,628 [MainThread] INFO UNet3DTrainer - Training iteration [2259/60000]. Epoch [40/499]
2023-02-03 15:36:21,183 [MainThread] INFO UNet3DTrainer - Training iteration [2260/60000]. Epoch [40/499]
2023-02-03 15:36:44,968 [MainThread] INFO UNet3DTrainer - Training iteration [2261/60000]. Epoch [40/499]
2023-02-03 15:36:45,371 [MainThread] INFO UNet3DTrainer - Training iteration [2262/60000]. Epoch [40/499]
2023-02-03 15:36:45,911 [MainThread] INFO UNet3DTrainer - Training iteration [2263/60000]. Epoch [40/499]
2023-02-03 15:36:46,448 [MainThread] INFO UNet3DTrainer - Training iteration [2264/60000]. Epoch [40/499]
2023-02-03 15:36:46,993 [MainThread] INFO UNet3DTrainer - Training iteration [2265/60000]. Epoch [40/499]
2023-02-03 15:36:47,545 [MainThread] INFO UNet3DTrainer - Training iteration [2266/60000]. Epoch [40/499]
2023-02-03 15:36:49,048 [MainThread] INFO UNet3DTrainer - Training iteration [2267/60000]. Epoch [40/499]
2023-02-03 15:36:49,530 [MainThread] INFO UNet3DTrainer - Training iteration [2268/60000]. Epoch [40/499]
2023-02-03 15:36:52,953 [MainThread] INFO UNet3DTrainer - Training iteration [2269/60000]. Epoch [40/499]
2023-02-03 15:37:08,118 [MainThread] INFO UNet3DTrainer - Training iteration [2270/60000]. Epoch [40/499]
2023-02-03 15:37:08,513 [MainThread] INFO UNet3DTrainer - Training iteration [2271/60000]. Epoch [40/499]
2023-02-03 15:37:09,063 [MainThread] INFO UNet3DTrainer - Training iteration [2272/60000]. Epoch [40/499]
2023-02-03 15:37:09,620 [MainThread] INFO UNet3DTrainer - Training iteration [2273/60000]. Epoch [40/499]
2023-02-03 15:37:13,260 [MainThread] INFO UNet3DTrainer - Training iteration [2274/60000]. Epoch [40/499]
2023-02-03 15:37:13,771 [MainThread] INFO UNet3DTrainer - Training iteration [2275/60000]. Epoch [40/499]
2023-02-03 15:37:54,712 [MainThread] INFO UNet3DTrainer - Training iteration [2276/60000]. Epoch [40/499]
2023-02-03 15:37:55,105 [MainThread] INFO UNet3DTrainer - Training iteration [2277/60000]. Epoch [40/499]
2023-02-03 15:37:55,632 [MainThread] INFO UNet3DTrainer - Training iteration [2278/60000]. Epoch [40/499]
2023-02-03 15:37:56,175 [MainThread] INFO UNet3DTrainer - Training iteration [2279/60000]. Epoch [40/499]
2023-02-03 15:37:56,734 [MainThread] INFO UNet3DTrainer - Training iteration [2280/60000]. Epoch [40/499]
2023-02-03 15:37:57,286 [MainThread] INFO UNet3DTrainer - Training iteration [2281/60000]. Epoch [40/499]
2023-02-03 15:37:59,072 [MainThread] INFO UNet3DTrainer - Training iteration [2282/60000]. Epoch [40/499]
2023-02-03 15:37:59,561 [MainThread] INFO UNet3DTrainer - Training iteration [2283/60000]. Epoch [40/499]
2023-02-03 15:38:00,122 [MainThread] INFO UNet3DTrainer - Training iteration [2284/60000]. Epoch [40/499]
2023-02-03 15:38:00,721 [MainThread] INFO UNet3DTrainer - Training iteration [2285/60000]. Epoch [40/499]
2023-02-03 15:38:01,255 [MainThread] INFO UNet3DTrainer - Training iteration [2286/60000]. Epoch [40/499]
2023-02-03 15:38:01,806 [MainThread] INFO UNet3DTrainer - Training iteration [2287/60000]. Epoch [40/499]
2023-02-03 15:38:25,109 [MainThread] INFO UNet3DTrainer - Training iteration [2288/60000]. Epoch [40/499]
2023-02-03 15:38:25,472 [MainThread] INFO UNet3DTrainer - Training iteration [2289/60000]. Epoch [40/499]
2023-02-03 15:38:26,004 [MainThread] INFO UNet3DTrainer - Training iteration [2290/60000]. Epoch [40/499]
2023-02-03 15:38:26,531 [MainThread] INFO UNet3DTrainer - Training iteration [2291/60000]. Epoch [40/499]
2023-02-03 15:38:27,073 [MainThread] INFO UNet3DTrainer - Training iteration [2292/60000]. Epoch [40/499]
2023-02-03 15:38:27,608 [MainThread] INFO UNet3DTrainer - Training iteration [2293/60000]. Epoch [40/499]
2023-02-03 15:38:28,139 [MainThread] INFO UNet3DTrainer - Training iteration [2294/60000]. Epoch [40/499]
2023-02-03 15:38:42,350 [MainThread] INFO UNet3DTrainer - Training iteration [2295/60000]. Epoch [40/499]
2023-02-03 15:38:42,723 [MainThread] INFO UNet3DTrainer - Training iteration [2296/60000]. Epoch [40/499]
2023-02-03 15:39:13,882 [MainThread] INFO UNet3DTrainer - Training iteration [2297/60000]. Epoch [41/499]
2023-02-03 15:39:14,940 [MainThread] INFO UNet3DTrainer - Training iteration [2298/60000]. Epoch [41/499]
2023-02-03 15:39:15,323 [MainThread] INFO UNet3DTrainer - Training iteration [2299/60000]. Epoch [41/499]
2023-02-03 15:39:15,883 [MainThread] INFO UNet3DTrainer - Training iteration [2300/60000]. Epoch [41/499]
2023-02-03 15:39:16,446 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 15:39:23,125 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 15:39:35,093 [MainThread] INFO EvalMetric - ARand: 0.6444292010769233
2023-02-03 15:39:35,142 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 15:39:44,783 [MainThread] INFO EvalMetric - ARand: 0.7057033924787568
2023-02-03 15:39:44,836 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 15:39:54,629 [MainThread] INFO EvalMetric - ARand: 0.7178523918756098
2023-02-03 15:39:54,689 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 15:40:02,601 [MainThread] INFO EvalMetric - ARand: 0.6752695131771436
2023-02-03 15:40:02,648 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 15:40:10,130 [MainThread] INFO EvalMetric - ARand: 0.667967024541525
2023-02-03 15:40:10,177 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 15:40:17,654 [MainThread] INFO EvalMetric - ARand: 0.6997774868059823
2023-02-03 15:40:17,703 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 15:40:25,169 [MainThread] INFO EvalMetric - ARand: 0.6764499250442223
2023-02-03 15:40:25,211 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 15:40:32,645 [MainThread] INFO EvalMetric - ARand: 0.7117971797845737
2023-02-03 15:40:32,687 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 15:40:40,077 [MainThread] INFO EvalMetric - ARand: 0.7244743949762428
2023-02-03 15:40:40,121 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 15:40:47,578 [MainThread] INFO EvalMetric - ARand: 0.6399572079171842
2023-02-03 15:40:47,616 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 15:40:55,111 [MainThread] INFO EvalMetric - ARand: 0.6303727677043323
2023-02-03 15:40:55,155 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 15:41:02,553 [MainThread] INFO EvalMetric - ARand: 0.6309822723192471
2023-02-03 15:41:02,592 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 15:41:10,044 [MainThread] INFO EvalMetric - ARand: 0.6756891186533311
2023-02-03 15:41:10,081 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 15:41:17,552 [MainThread] INFO EvalMetric - ARand: 0.7056763786583702
2023-02-03 15:41:17,591 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 15:41:25,020 [MainThread] INFO EvalMetric - ARand: 0.6962920403050782
2023-02-03 15:41:25,064 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 15:41:32,518 [MainThread] INFO EvalMetric - ARand: 0.6688456504619033
2023-02-03 15:41:32,569 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 15:41:40,119 [MainThread] INFO EvalMetric - ARand: 0.6523123045717896
2023-02-03 15:41:40,161 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 15:41:47,563 [MainThread] INFO EvalMetric - ARand: 0.7061330344660965
2023-02-03 15:41:47,608 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 15:41:55,119 [MainThread] INFO EvalMetric - ARand: 0.6769264959055326
2023-02-03 15:41:55,171 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 15:42:02,741 [MainThread] INFO EvalMetric - ARand: 0.6823828408291457
2023-02-03 15:42:02,787 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 15:42:10,296 [MainThread] INFO EvalMetric - ARand: 0.7562223668725678
2023-02-03 15:42:10,340 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 15:42:16,005 [MainThread] INFO EvalMetric - ARand: 0.7580656316783129
2023-02-03 15:42:16,393 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4763937722677472. Evaluation score: 0.6857040097555996
2023-02-03 15:42:16,416 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 15:42:23,799 [MainThread] INFO EvalMetric - ARand: 0.7926046694750968
2023-02-03 15:42:23,820 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.13155876472592354. Evaluation score: 0.7926046694750968
2023-02-03 15:42:23,821 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 15:42:24,885 [MainThread] INFO UNet3DTrainer - Training iteration [2301/60000]. Epoch [41/499]
2023-02-03 15:42:25,270 [MainThread] INFO UNet3DTrainer - Training iteration [2302/60000]. Epoch [41/499]
2023-02-03 15:42:25,818 [MainThread] INFO UNet3DTrainer - Training iteration [2303/60000]. Epoch [41/499]
2023-02-03 15:42:26,350 [MainThread] INFO UNet3DTrainer - Training iteration [2304/60000]. Epoch [41/499]
2023-02-03 15:42:26,895 [MainThread] INFO UNet3DTrainer - Training iteration [2305/60000]. Epoch [41/499]
2023-02-03 15:42:27,462 [MainThread] INFO UNet3DTrainer - Training iteration [2306/60000]. Epoch [41/499]
2023-02-03 15:42:28,130 [MainThread] INFO UNet3DTrainer - Training iteration [2307/60000]. Epoch [41/499]
2023-02-03 15:42:28,796 [MainThread] INFO UNet3DTrainer - Training iteration [2308/60000]. Epoch [41/499]
2023-02-03 15:42:29,400 [MainThread] INFO UNet3DTrainer - Training iteration [2309/60000]. Epoch [41/499]
2023-02-03 15:42:29,971 [MainThread] INFO UNet3DTrainer - Training iteration [2310/60000]. Epoch [41/499]
2023-02-03 15:42:30,566 [MainThread] INFO UNet3DTrainer - Training iteration [2311/60000]. Epoch [41/499]
2023-02-03 15:42:31,290 [MainThread] INFO UNet3DTrainer - Training iteration [2312/60000]. Epoch [41/499]
2023-02-03 15:42:31,841 [MainThread] INFO UNet3DTrainer - Training iteration [2313/60000]. Epoch [41/499]
2023-02-03 15:42:32,484 [MainThread] INFO UNet3DTrainer - Training iteration [2314/60000]. Epoch [41/499]
2023-02-03 15:42:33,100 [MainThread] INFO UNet3DTrainer - Training iteration [2315/60000]. Epoch [41/499]
2023-02-03 15:42:33,672 [MainThread] INFO UNet3DTrainer - Training iteration [2316/60000]. Epoch [41/499]
2023-02-03 15:42:34,259 [MainThread] INFO UNet3DTrainer - Training iteration [2317/60000]. Epoch [41/499]
2023-02-03 15:42:34,850 [MainThread] INFO UNet3DTrainer - Training iteration [2318/60000]. Epoch [41/499]
2023-02-03 15:42:35,450 [MainThread] INFO UNet3DTrainer - Training iteration [2319/60000]. Epoch [41/499]
2023-02-03 15:42:36,080 [MainThread] INFO UNet3DTrainer - Training iteration [2320/60000]. Epoch [41/499]
2023-02-03 15:43:00,744 [MainThread] INFO UNet3DTrainer - Training iteration [2321/60000]. Epoch [41/499]
2023-02-03 15:43:01,170 [MainThread] INFO UNet3DTrainer - Training iteration [2322/60000]. Epoch [41/499]
2023-02-03 15:43:01,726 [MainThread] INFO UNet3DTrainer - Training iteration [2323/60000]. Epoch [41/499]
2023-02-03 15:43:02,297 [MainThread] INFO UNet3DTrainer - Training iteration [2324/60000]. Epoch [41/499]
2023-02-03 15:43:03,681 [MainThread] INFO UNet3DTrainer - Training iteration [2325/60000]. Epoch [41/499]
2023-02-03 15:43:04,056 [MainThread] INFO UNet3DTrainer - Training iteration [2326/60000]. Epoch [41/499]
2023-02-03 15:43:05,680 [MainThread] INFO UNet3DTrainer - Training iteration [2327/60000]. Epoch [41/499]
2023-02-03 15:43:06,183 [MainThread] INFO UNet3DTrainer - Training iteration [2328/60000]. Epoch [41/499]
2023-02-03 15:43:49,111 [MainThread] INFO UNet3DTrainer - Training iteration [2329/60000]. Epoch [41/499]
2023-02-03 15:43:49,481 [MainThread] INFO UNet3DTrainer - Training iteration [2330/60000]. Epoch [41/499]
2023-02-03 15:43:50,026 [MainThread] INFO UNet3DTrainer - Training iteration [2331/60000]. Epoch [41/499]
2023-02-03 15:43:50,576 [MainThread] INFO UNet3DTrainer - Training iteration [2332/60000]. Epoch [41/499]
2023-02-03 15:43:51,134 [MainThread] INFO UNet3DTrainer - Training iteration [2333/60000]. Epoch [41/499]
2023-02-03 15:43:51,700 [MainThread] INFO UNet3DTrainer - Training iteration [2334/60000]. Epoch [41/499]
2023-02-03 15:43:54,276 [MainThread] INFO UNet3DTrainer - Training iteration [2335/60000]. Epoch [41/499]
2023-02-03 15:43:54,727 [MainThread] INFO UNet3DTrainer - Training iteration [2336/60000]. Epoch [41/499]
2023-02-03 15:43:55,301 [MainThread] INFO UNet3DTrainer - Training iteration [2337/60000]. Epoch [41/499]
2023-02-03 15:43:55,861 [MainThread] INFO UNet3DTrainer - Training iteration [2338/60000]. Epoch [41/499]
2023-02-03 15:43:58,283 [MainThread] INFO UNet3DTrainer - Training iteration [2339/60000]. Epoch [41/499]
2023-02-03 15:43:58,650 [MainThread] INFO UNet3DTrainer - Training iteration [2340/60000]. Epoch [41/499]
2023-02-03 15:44:24,479 [MainThread] INFO UNet3DTrainer - Training iteration [2341/60000]. Epoch [41/499]
2023-02-03 15:44:25,000 [MainThread] INFO UNet3DTrainer - Training iteration [2342/60000]. Epoch [41/499]
2023-02-03 15:44:25,580 [MainThread] INFO UNet3DTrainer - Training iteration [2343/60000]. Epoch [41/499]
2023-02-03 15:44:26,116 [MainThread] INFO UNet3DTrainer - Training iteration [2344/60000]. Epoch [41/499]
2023-02-03 15:44:46,054 [MainThread] INFO UNet3DTrainer - Training iteration [2345/60000]. Epoch [41/499]
2023-02-03 15:44:46,440 [MainThread] INFO UNet3DTrainer - Training iteration [2346/60000]. Epoch [41/499]
2023-02-03 15:44:46,975 [MainThread] INFO UNet3DTrainer - Training iteration [2347/60000]. Epoch [41/499]
2023-02-03 15:44:47,505 [MainThread] INFO UNet3DTrainer - Training iteration [2348/60000]. Epoch [41/499]
2023-02-03 15:44:48,038 [MainThread] INFO UNet3DTrainer - Training iteration [2349/60000]. Epoch [41/499]
2023-02-03 15:44:48,575 [MainThread] INFO UNet3DTrainer - Training iteration [2350/60000]. Epoch [41/499]
2023-02-03 15:45:07,197 [MainThread] INFO UNet3DTrainer - Training iteration [2351/60000]. Epoch [41/499]
2023-02-03 15:45:07,580 [MainThread] INFO UNet3DTrainer - Training iteration [2352/60000]. Epoch [41/499]
2023-02-03 15:45:15,414 [MainThread] INFO UNet3DTrainer - Training iteration [2353/60000]. Epoch [42/499]
2023-02-03 15:45:15,970 [MainThread] INFO UNet3DTrainer - Training iteration [2354/60000]. Epoch [42/499]
2023-02-03 15:45:38,289 [MainThread] INFO UNet3DTrainer - Training iteration [2355/60000]. Epoch [42/499]
2023-02-03 15:45:38,687 [MainThread] INFO UNet3DTrainer - Training iteration [2356/60000]. Epoch [42/499]
2023-02-03 15:45:39,257 [MainThread] INFO UNet3DTrainer - Training iteration [2357/60000]. Epoch [42/499]
2023-02-03 15:45:39,814 [MainThread] INFO UNet3DTrainer - Training iteration [2358/60000]. Epoch [42/499]
2023-02-03 15:45:42,040 [MainThread] INFO UNet3DTrainer - Training iteration [2359/60000]. Epoch [42/499]
2023-02-03 15:45:42,530 [MainThread] INFO UNet3DTrainer - Training iteration [2360/60000]. Epoch [42/499]
2023-02-03 15:46:02,435 [MainThread] INFO UNet3DTrainer - Training iteration [2361/60000]. Epoch [42/499]
2023-02-03 15:46:02,824 [MainThread] INFO UNet3DTrainer - Training iteration [2362/60000]. Epoch [42/499]
2023-02-03 15:46:03,387 [MainThread] INFO UNet3DTrainer - Training iteration [2363/60000]. Epoch [42/499]
2023-02-03 15:46:03,928 [MainThread] INFO UNet3DTrainer - Training iteration [2364/60000]. Epoch [42/499]
2023-02-03 15:46:04,501 [MainThread] INFO UNet3DTrainer - Training iteration [2365/60000]. Epoch [42/499]
2023-02-03 15:46:05,585 [MainThread] INFO UNet3DTrainer - Training iteration [2366/60000]. Epoch [42/499]
2023-02-03 15:46:06,845 [MainThread] INFO UNet3DTrainer - Training iteration [2367/60000]. Epoch [42/499]
2023-02-03 15:46:07,270 [MainThread] INFO UNet3DTrainer - Training iteration [2368/60000]. Epoch [42/499]
2023-02-03 15:46:07,806 [MainThread] INFO UNet3DTrainer - Training iteration [2369/60000]. Epoch [42/499]
2023-02-03 15:46:08,474 [MainThread] INFO UNet3DTrainer - Training iteration [2370/60000]. Epoch [42/499]
2023-02-03 15:46:09,048 [MainThread] INFO UNet3DTrainer - Training iteration [2371/60000]. Epoch [42/499]
2023-02-03 15:46:35,981 [MainThread] INFO UNet3DTrainer - Training iteration [2372/60000]. Epoch [42/499]
2023-02-03 15:46:37,210 [MainThread] INFO UNet3DTrainer - Training iteration [2373/60000]. Epoch [42/499]
2023-02-03 15:46:37,703 [MainThread] INFO UNet3DTrainer - Training iteration [2374/60000]. Epoch [42/499]
2023-02-03 15:46:38,344 [MainThread] INFO UNet3DTrainer - Training iteration [2375/60000]. Epoch [42/499]
2023-02-03 15:46:59,911 [MainThread] INFO UNet3DTrainer - Training iteration [2376/60000]. Epoch [42/499]
2023-02-03 15:47:00,307 [MainThread] INFO UNet3DTrainer - Training iteration [2377/60000]. Epoch [42/499]
2023-02-03 15:47:04,436 [MainThread] INFO UNet3DTrainer - Training iteration [2378/60000]. Epoch [42/499]
2023-02-03 15:47:27,533 [MainThread] INFO UNet3DTrainer - Training iteration [2379/60000]. Epoch [42/499]
2023-02-03 15:47:27,945 [MainThread] INFO UNet3DTrainer - Training iteration [2380/60000]. Epoch [42/499]
2023-02-03 15:47:28,492 [MainThread] INFO UNet3DTrainer - Training iteration [2381/60000]. Epoch [42/499]
2023-02-03 15:47:29,057 [MainThread] INFO UNet3DTrainer - Training iteration [2382/60000]. Epoch [42/499]
2023-02-03 15:47:29,605 [MainThread] INFO UNet3DTrainer - Training iteration [2383/60000]. Epoch [42/499]
2023-02-03 15:47:30,190 [MainThread] INFO UNet3DTrainer - Training iteration [2384/60000]. Epoch [42/499]
2023-02-03 15:48:09,493 [MainThread] INFO UNet3DTrainer - Training iteration [2385/60000]. Epoch [42/499]
2023-02-03 15:48:09,867 [MainThread] INFO UNet3DTrainer - Training iteration [2386/60000]. Epoch [42/499]
2023-02-03 15:48:10,398 [MainThread] INFO UNet3DTrainer - Training iteration [2387/60000]. Epoch [42/499]
2023-02-03 15:48:10,955 [MainThread] INFO UNet3DTrainer - Training iteration [2388/60000]. Epoch [42/499]
2023-02-03 15:48:11,509 [MainThread] INFO UNet3DTrainer - Training iteration [2389/60000]. Epoch [42/499]
2023-02-03 15:48:12,057 [MainThread] INFO UNet3DTrainer - Training iteration [2390/60000]. Epoch [42/499]
2023-02-03 15:48:32,208 [MainThread] INFO UNet3DTrainer - Training iteration [2391/60000]. Epoch [42/499]
2023-02-03 15:48:32,598 [MainThread] INFO UNet3DTrainer - Training iteration [2392/60000]. Epoch [42/499]
2023-02-03 15:48:33,156 [MainThread] INFO UNet3DTrainer - Training iteration [2393/60000]. Epoch [42/499]
2023-02-03 15:48:33,717 [MainThread] INFO UNet3DTrainer - Training iteration [2394/60000]. Epoch [42/499]
2023-02-03 15:48:34,259 [MainThread] INFO UNet3DTrainer - Training iteration [2395/60000]. Epoch [42/499]
2023-02-03 15:48:34,850 [MainThread] INFO UNet3DTrainer - Training iteration [2396/60000]. Epoch [42/499]
2023-02-03 15:48:36,358 [MainThread] INFO UNet3DTrainer - Training iteration [2397/60000]. Epoch [42/499]
2023-02-03 15:48:36,822 [MainThread] INFO UNet3DTrainer - Training iteration [2398/60000]. Epoch [42/499]
2023-02-03 15:48:37,377 [MainThread] INFO UNet3DTrainer - Training iteration [2399/60000]. Epoch [42/499]
2023-02-03 15:48:53,376 [MainThread] INFO UNet3DTrainer - Training iteration [2400/60000]. Epoch [42/499]
2023-02-03 15:48:53,739 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 15:48:59,502 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 15:49:08,258 [MainThread] INFO EvalMetric - ARand: 0.6658705187089872
2023-02-03 15:49:08,300 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 15:49:15,681 [MainThread] INFO EvalMetric - ARand: 0.6810319576792919
2023-02-03 15:49:15,722 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 15:49:23,029 [MainThread] INFO EvalMetric - ARand: 0.7095965544454461
2023-02-03 15:49:23,065 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 15:49:30,474 [MainThread] INFO EvalMetric - ARand: 0.6269654307261985
2023-02-03 15:49:30,516 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 15:49:37,853 [MainThread] INFO EvalMetric - ARand: 0.6812601640358417
2023-02-03 15:49:37,902 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 15:49:45,214 [MainThread] INFO EvalMetric - ARand: 0.6997128584848337
2023-02-03 15:49:45,264 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 15:49:52,618 [MainThread] INFO EvalMetric - ARand: 0.626901468663095
2023-02-03 15:49:52,663 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 15:50:00,132 [MainThread] INFO EvalMetric - ARand: 0.7311716676835067
2023-02-03 15:50:00,168 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 15:50:07,588 [MainThread] INFO EvalMetric - ARand: 0.7039319116200998
2023-02-03 15:50:07,626 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 15:50:15,102 [MainThread] INFO EvalMetric - ARand: 0.5945123482278043
2023-02-03 15:50:15,145 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 15:50:22,628 [MainThread] INFO EvalMetric - ARand: 0.5790963044849643
2023-02-03 15:50:22,666 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 15:50:30,130 [MainThread] INFO EvalMetric - ARand: 0.6241635181565176
2023-02-03 15:50:30,173 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 15:50:37,505 [MainThread] INFO EvalMetric - ARand: 0.594856702285405
2023-02-03 15:50:37,542 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 15:50:44,900 [MainThread] INFO EvalMetric - ARand: 0.7290013090946701
2023-02-03 15:50:44,948 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 15:50:52,342 [MainThread] INFO EvalMetric - ARand: 0.6808342005518269
2023-02-03 15:50:52,383 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 15:50:59,721 [MainThread] INFO EvalMetric - ARand: 0.66523793371801
2023-02-03 15:50:59,764 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 15:51:07,270 [MainThread] INFO EvalMetric - ARand: 0.5914876430413891
2023-02-03 15:51:07,313 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 15:51:14,743 [MainThread] INFO EvalMetric - ARand: 0.6786423078377518
2023-02-03 15:51:14,787 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 15:51:22,261 [MainThread] INFO EvalMetric - ARand: 0.6642676289415574
2023-02-03 15:51:22,305 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 15:51:29,779 [MainThread] INFO EvalMetric - ARand: 0.6158029734222538
2023-02-03 15:51:29,825 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 15:51:37,305 [MainThread] INFO EvalMetric - ARand: 0.7166297163914528
2023-02-03 15:51:37,349 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 15:51:43,033 [MainThread] INFO EvalMetric - ARand: 0.6922279332912754
2023-02-03 15:51:43,423 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4745878118208084. Evaluation score: 0.6611561410652579
2023-02-03 15:51:43,447 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 15:51:50,689 [MainThread] INFO EvalMetric - ARand: 0.7546505443063276
2023-02-03 15:51:50,709 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.12621317943558097. Evaluation score: 0.7546505443063276
2023-02-03 15:51:50,710 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 15:51:51,764 [MainThread] INFO UNet3DTrainer - Training iteration [2401/60000]. Epoch [42/499]
2023-02-03 15:51:52,138 [MainThread] INFO UNet3DTrainer - Training iteration [2402/60000]. Epoch [42/499]
2023-02-03 15:51:52,665 [MainThread] INFO UNet3DTrainer - Training iteration [2403/60000]. Epoch [42/499]
2023-02-03 15:51:53,192 [MainThread] INFO UNet3DTrainer - Training iteration [2404/60000]. Epoch [42/499]
2023-02-03 15:51:53,722 [MainThread] INFO UNet3DTrainer - Training iteration [2405/60000]. Epoch [42/499]
2023-02-03 15:51:54,257 [MainThread] INFO UNet3DTrainer - Training iteration [2406/60000]. Epoch [42/499]
2023-02-03 15:51:54,789 [MainThread] INFO UNet3DTrainer - Training iteration [2407/60000]. Epoch [42/499]
2023-02-03 15:51:55,316 [MainThread] INFO UNet3DTrainer - Training iteration [2408/60000]. Epoch [42/499]
2023-02-03 15:52:26,434 [MainThread] INFO UNet3DTrainer - Training iteration [2409/60000]. Epoch [43/499]
2023-02-03 15:52:26,854 [MainThread] INFO UNet3DTrainer - Training iteration [2410/60000]. Epoch [43/499]
2023-02-03 15:52:27,448 [MainThread] INFO UNet3DTrainer - Training iteration [2411/60000]. Epoch [43/499]
2023-02-03 15:52:46,972 [MainThread] INFO UNet3DTrainer - Training iteration [2412/60000]. Epoch [43/499]
2023-02-03 15:52:47,364 [MainThread] INFO UNet3DTrainer - Training iteration [2413/60000]. Epoch [43/499]
2023-02-03 15:52:47,896 [MainThread] INFO UNet3DTrainer - Training iteration [2414/60000]. Epoch [43/499]
2023-02-03 15:53:08,448 [MainThread] INFO UNet3DTrainer - Training iteration [2415/60000]. Epoch [43/499]
2023-02-03 15:53:08,820 [MainThread] INFO UNet3DTrainer - Training iteration [2416/60000]. Epoch [43/499]
2023-02-03 15:53:09,370 [MainThread] INFO UNet3DTrainer - Training iteration [2417/60000]. Epoch [43/499]
2023-02-03 15:53:28,153 [MainThread] INFO UNet3DTrainer - Training iteration [2418/60000]. Epoch [43/499]
2023-02-03 15:53:28,539 [MainThread] INFO UNet3DTrainer - Training iteration [2419/60000]. Epoch [43/499]
2023-02-03 15:53:29,086 [MainThread] INFO UNet3DTrainer - Training iteration [2420/60000]. Epoch [43/499]
2023-02-03 15:53:29,634 [MainThread] INFO UNet3DTrainer - Training iteration [2421/60000]. Epoch [43/499]
2023-02-03 15:53:30,195 [MainThread] INFO UNet3DTrainer - Training iteration [2422/60000]. Epoch [43/499]
2023-02-03 15:53:30,751 [MainThread] INFO UNet3DTrainer - Training iteration [2423/60000]. Epoch [43/499]
2023-02-03 15:53:33,071 [MainThread] INFO UNet3DTrainer - Training iteration [2424/60000]. Epoch [43/499]
2023-02-03 15:53:33,534 [MainThread] INFO UNet3DTrainer - Training iteration [2425/60000]. Epoch [43/499]
2023-02-03 15:53:34,193 [MainThread] INFO UNet3DTrainer - Training iteration [2426/60000]. Epoch [43/499]
2023-02-03 15:53:34,750 [MainThread] INFO UNet3DTrainer - Training iteration [2427/60000]. Epoch [43/499]
2023-02-03 15:53:35,350 [MainThread] INFO UNet3DTrainer - Training iteration [2428/60000]. Epoch [43/499]
2023-02-03 15:53:35,918 [MainThread] INFO UNet3DTrainer - Training iteration [2429/60000]. Epoch [43/499]
2023-02-03 15:53:37,831 [MainThread] INFO UNet3DTrainer - Training iteration [2430/60000]. Epoch [43/499]
2023-02-03 15:53:57,469 [MainThread] INFO UNet3DTrainer - Training iteration [2431/60000]. Epoch [43/499]
2023-02-03 15:53:57,859 [MainThread] INFO UNet3DTrainer - Training iteration [2432/60000]. Epoch [43/499]
2023-02-03 15:53:58,424 [MainThread] INFO UNet3DTrainer - Training iteration [2433/60000]. Epoch [43/499]
2023-02-03 15:54:21,631 [MainThread] INFO UNet3DTrainer - Training iteration [2434/60000]. Epoch [43/499]
2023-02-03 15:54:22,005 [MainThread] INFO UNet3DTrainer - Training iteration [2435/60000]. Epoch [43/499]
2023-02-03 15:54:22,565 [MainThread] INFO UNet3DTrainer - Training iteration [2436/60000]. Epoch [43/499]
2023-02-03 15:54:23,077 [MainThread] INFO UNet3DTrainer - Training iteration [2437/60000]. Epoch [43/499]
2023-02-03 15:54:23,633 [MainThread] INFO UNet3DTrainer - Training iteration [2438/60000]. Epoch [43/499]
2023-02-03 15:54:24,191 [MainThread] INFO UNet3DTrainer - Training iteration [2439/60000]. Epoch [43/499]
2023-02-03 15:54:44,615 [MainThread] INFO UNet3DTrainer - Training iteration [2440/60000]. Epoch [43/499]
2023-02-03 15:54:44,989 [MainThread] INFO UNet3DTrainer - Training iteration [2441/60000]. Epoch [43/499]
2023-02-03 15:54:45,547 [MainThread] INFO UNet3DTrainer - Training iteration [2442/60000]. Epoch [43/499]
2023-02-03 15:54:46,074 [MainThread] INFO UNet3DTrainer - Training iteration [2443/60000]. Epoch [43/499]
2023-02-03 15:54:46,638 [MainThread] INFO UNet3DTrainer - Training iteration [2444/60000]. Epoch [43/499]
2023-02-03 15:54:47,337 [MainThread] INFO UNet3DTrainer - Training iteration [2445/60000]. Epoch [43/499]
2023-02-03 15:55:07,343 [MainThread] INFO UNet3DTrainer - Training iteration [2446/60000]. Epoch [43/499]
2023-02-03 15:55:07,715 [MainThread] INFO UNet3DTrainer - Training iteration [2447/60000]. Epoch [43/499]
2023-02-03 15:55:08,251 [MainThread] INFO UNet3DTrainer - Training iteration [2448/60000]. Epoch [43/499]
2023-02-03 15:55:08,807 [MainThread] INFO UNet3DTrainer - Training iteration [2449/60000]. Epoch [43/499]
2023-02-03 15:55:09,359 [MainThread] INFO UNet3DTrainer - Training iteration [2450/60000]. Epoch [43/499]
2023-02-03 15:55:09,907 [MainThread] INFO UNet3DTrainer - Training iteration [2451/60000]. Epoch [43/499]
2023-02-03 15:55:10,863 [MainThread] INFO UNet3DTrainer - Training iteration [2452/60000]. Epoch [43/499]
2023-02-03 15:55:33,627 [MainThread] INFO UNet3DTrainer - Training iteration [2453/60000]. Epoch [43/499]
2023-02-03 15:55:34,054 [MainThread] INFO UNet3DTrainer - Training iteration [2454/60000]. Epoch [43/499]
2023-02-03 15:55:34,603 [MainThread] INFO UNet3DTrainer - Training iteration [2455/60000]. Epoch [43/499]
2023-02-03 15:55:35,156 [MainThread] INFO UNet3DTrainer - Training iteration [2456/60000]. Epoch [43/499]
2023-02-03 15:55:35,721 [MainThread] INFO UNet3DTrainer - Training iteration [2457/60000]. Epoch [43/499]
2023-02-03 15:55:39,178 [MainThread] INFO UNet3DTrainer - Training iteration [2458/60000]. Epoch [43/499]
2023-02-03 15:55:39,562 [MainThread] INFO UNet3DTrainer - Training iteration [2459/60000]. Epoch [43/499]
2023-02-03 15:55:40,091 [MainThread] INFO UNet3DTrainer - Training iteration [2460/60000]. Epoch [43/499]
2023-02-03 15:55:55,898 [MainThread] INFO UNet3DTrainer - Training iteration [2461/60000]. Epoch [43/499]
2023-02-03 15:55:56,274 [MainThread] INFO UNet3DTrainer - Training iteration [2462/60000]. Epoch [43/499]
2023-02-03 15:55:56,807 [MainThread] INFO UNet3DTrainer - Training iteration [2463/60000]. Epoch [43/499]
2023-02-03 15:55:57,336 [MainThread] INFO UNet3DTrainer - Training iteration [2464/60000]. Epoch [43/499]
2023-02-03 15:56:26,319 [MainThread] INFO UNet3DTrainer - Training iteration [2465/60000]. Epoch [44/499]
2023-02-03 15:56:27,853 [MainThread] INFO UNet3DTrainer - Training iteration [2466/60000]. Epoch [44/499]
2023-02-03 15:56:28,203 [MainThread] INFO UNet3DTrainer - Training iteration [2467/60000]. Epoch [44/499]
2023-02-03 15:56:28,764 [MainThread] INFO UNet3DTrainer - Training iteration [2468/60000]. Epoch [44/499]
2023-02-03 15:56:29,366 [MainThread] INFO UNet3DTrainer - Training iteration [2469/60000]. Epoch [44/499]
2023-02-03 15:56:29,930 [MainThread] INFO UNet3DTrainer - Training iteration [2470/60000]. Epoch [44/499]
2023-02-03 15:56:52,161 [MainThread] INFO UNet3DTrainer - Training iteration [2471/60000]. Epoch [44/499]
2023-02-03 15:56:54,362 [MainThread] INFO UNet3DTrainer - Training iteration [2472/60000]. Epoch [44/499]
2023-02-03 15:56:54,736 [MainThread] INFO UNet3DTrainer - Training iteration [2473/60000]. Epoch [44/499]
2023-02-03 15:56:55,307 [MainThread] INFO UNet3DTrainer - Training iteration [2474/60000]. Epoch [44/499]
2023-02-03 15:56:55,860 [MainThread] INFO UNet3DTrainer - Training iteration [2475/60000]. Epoch [44/499]
2023-02-03 15:56:56,542 [MainThread] INFO UNet3DTrainer - Training iteration [2476/60000]. Epoch [44/499]
2023-02-03 15:56:57,110 [MainThread] INFO UNet3DTrainer - Training iteration [2477/60000]. Epoch [44/499]
2023-02-03 15:57:20,871 [MainThread] INFO UNet3DTrainer - Training iteration [2478/60000]. Epoch [44/499]
2023-02-03 15:57:21,271 [MainThread] INFO UNet3DTrainer - Training iteration [2479/60000]. Epoch [44/499]
2023-02-03 15:57:21,809 [MainThread] INFO UNet3DTrainer - Training iteration [2480/60000]. Epoch [44/499]
2023-02-03 15:57:22,350 [MainThread] INFO UNet3DTrainer - Training iteration [2481/60000]. Epoch [44/499]
2023-02-03 15:57:22,908 [MainThread] INFO UNet3DTrainer - Training iteration [2482/60000]. Epoch [44/499]
2023-02-03 15:57:47,611 [MainThread] INFO UNet3DTrainer - Training iteration [2483/60000]. Epoch [44/499]
2023-02-03 15:57:51,298 [MainThread] INFO UNet3DTrainer - Training iteration [2484/60000]. Epoch [44/499]
2023-02-03 15:57:51,761 [MainThread] INFO UNet3DTrainer - Training iteration [2485/60000]. Epoch [44/499]
2023-02-03 15:57:52,297 [MainThread] INFO UNet3DTrainer - Training iteration [2486/60000]. Epoch [44/499]
2023-02-03 15:57:52,950 [MainThread] INFO UNet3DTrainer - Training iteration [2487/60000]. Epoch [44/499]
2023-02-03 15:57:53,488 [MainThread] INFO UNet3DTrainer - Training iteration [2488/60000]. Epoch [44/499]
2023-02-03 15:57:54,122 [MainThread] INFO UNet3DTrainer - Training iteration [2489/60000]. Epoch [44/499]
2023-02-03 15:58:18,377 [MainThread] INFO UNet3DTrainer - Training iteration [2490/60000]. Epoch [44/499]
2023-02-03 15:58:18,781 [MainThread] INFO UNet3DTrainer - Training iteration [2491/60000]. Epoch [44/499]
2023-02-03 15:58:19,313 [MainThread] INFO UNet3DTrainer - Training iteration [2492/60000]. Epoch [44/499]
2023-02-03 15:58:19,861 [MainThread] INFO UNet3DTrainer - Training iteration [2493/60000]. Epoch [44/499]
2023-02-03 15:58:20,412 [MainThread] INFO UNet3DTrainer - Training iteration [2494/60000]. Epoch [44/499]
2023-02-03 15:58:20,977 [MainThread] INFO UNet3DTrainer - Training iteration [2495/60000]. Epoch [44/499]
2023-02-03 15:58:22,560 [MainThread] INFO UNet3DTrainer - Training iteration [2496/60000]. Epoch [44/499]
2023-02-03 15:58:23,080 [MainThread] INFO UNet3DTrainer - Training iteration [2497/60000]. Epoch [44/499]
2023-02-03 15:58:23,681 [MainThread] INFO UNet3DTrainer - Training iteration [2498/60000]. Epoch [44/499]
2023-02-03 15:58:24,350 [MainThread] INFO UNet3DTrainer - Training iteration [2499/60000]. Epoch [44/499]
2023-02-03 15:58:24,902 [MainThread] INFO UNet3DTrainer - Training iteration [2500/60000]. Epoch [44/499]
2023-02-03 15:58:25,428 [MainThread] INFO UNet3DTrainer - Validating...
2023-02-03 15:58:31,928 [MainThread] INFO UNet3DTrainer - Validation iteration 0
2023-02-03 15:58:44,170 [MainThread] INFO EvalMetric - ARand: 0.6646445345941562
2023-02-03 15:58:44,217 [MainThread] INFO UNet3DTrainer - Validation iteration 1
2023-02-03 15:58:54,306 [MainThread] INFO EvalMetric - ARand: 0.6899339884470497
2023-02-03 15:58:54,354 [MainThread] INFO UNet3DTrainer - Validation iteration 2
2023-02-03 15:59:03,431 [MainThread] INFO EvalMetric - ARand: 0.7628635675722486
2023-02-03 15:59:03,474 [MainThread] INFO UNet3DTrainer - Validation iteration 3
2023-02-03 15:59:11,905 [MainThread] INFO EvalMetric - ARand: 0.6843773464688905
2023-02-03 15:59:11,947 [MainThread] INFO UNet3DTrainer - Validation iteration 4
2023-02-03 15:59:20,242 [MainThread] INFO EvalMetric - ARand: 0.6875338685466919
2023-02-03 15:59:20,299 [MainThread] INFO UNet3DTrainer - Validation iteration 5
2023-02-03 15:59:28,376 [MainThread] INFO EvalMetric - ARand: 0.7213486537389577
2023-02-03 15:59:28,426 [MainThread] INFO UNet3DTrainer - Validation iteration 6
2023-02-03 15:59:36,008 [MainThread] INFO EvalMetric - ARand: 0.6502150353275091
2023-02-03 15:59:36,057 [MainThread] INFO UNet3DTrainer - Validation iteration 7
2023-02-03 15:59:43,509 [MainThread] INFO EvalMetric - ARand: 0.7641644031902531
2023-02-03 15:59:43,549 [MainThread] INFO UNet3DTrainer - Validation iteration 8
2023-02-03 15:59:51,016 [MainThread] INFO EvalMetric - ARand: 0.7302346840526228
2023-02-03 15:59:51,056 [MainThread] INFO UNet3DTrainer - Validation iteration 9
2023-02-03 15:59:58,664 [MainThread] INFO EvalMetric - ARand: 0.6096233392561843
2023-02-03 15:59:58,708 [MainThread] INFO UNet3DTrainer - Validation iteration 10
2023-02-03 16:00:06,206 [MainThread] INFO EvalMetric - ARand: 0.5887622730002937
2023-02-03 16:00:06,248 [MainThread] INFO UNet3DTrainer - Validation iteration 11
2023-02-03 16:00:13,825 [MainThread] INFO EvalMetric - ARand: 0.6268069088113075
2023-02-03 16:00:13,868 [MainThread] INFO UNet3DTrainer - Validation iteration 12
2023-02-03 16:00:21,333 [MainThread] INFO EvalMetric - ARand: 0.6609734649500585
2023-02-03 16:00:21,373 [MainThread] INFO UNet3DTrainer - Validation iteration 13
2023-02-03 16:00:28,934 [MainThread] INFO EvalMetric - ARand: 0.7413471916483871
2023-02-03 16:00:28,978 [MainThread] INFO UNet3DTrainer - Validation iteration 14
2023-02-03 16:00:36,502 [MainThread] INFO EvalMetric - ARand: 0.6809213451972024
2023-02-03 16:00:36,541 [MainThread] INFO UNet3DTrainer - Validation iteration 15
2023-02-03 16:00:44,054 [MainThread] INFO EvalMetric - ARand: 0.6882595028085177
2023-02-03 16:00:44,092 [MainThread] INFO UNet3DTrainer - Validation iteration 16
2023-02-03 16:00:51,611 [MainThread] INFO EvalMetric - ARand: 0.6152993444203292
2023-02-03 16:00:51,655 [MainThread] INFO UNet3DTrainer - Validation iteration 17
2023-02-03 16:00:59,165 [MainThread] INFO EvalMetric - ARand: 0.7235574837426371
2023-02-03 16:00:59,206 [MainThread] INFO UNet3DTrainer - Validation iteration 18
2023-02-03 16:01:06,751 [MainThread] INFO EvalMetric - ARand: 0.7020173276117072
2023-02-03 16:01:06,797 [MainThread] INFO UNet3DTrainer - Validation iteration 19
2023-02-03 16:01:14,409 [MainThread] INFO EvalMetric - ARand: 0.660235512104886
2023-02-03 16:01:14,462 [MainThread] INFO UNet3DTrainer - Validation iteration 20
2023-02-03 16:01:22,128 [MainThread] INFO EvalMetric - ARand: 0.7383462385889609
2023-02-03 16:01:22,175 [MainThread] INFO UNet3DTrainer - Validation iteration 21
2023-02-03 16:01:27,858 [MainThread] INFO EvalMetric - ARand: 0.7545460445660428
2023-02-03 16:01:28,269 [MainThread] INFO UNet3DTrainer - Validation finished. Loss: 1.4739282651879322. Evaluation score: 0.68769542747142
2023-02-03 16:01:28,293 [MainThread] INFO UNet3DTrainer - Saving checkpoint to 'Model_3/last_checkpoint.pytorch'
2023-02-03 16:01:35,605 [MainThread] INFO EvalMetric - ARand: 0.846036648691568
2023-02-03 16:01:35,626 [MainThread] INFO UNet3DTrainer - Training stats. Loss: 0.11591588560905722. Evaluation score: 0.846036648691568
2023-02-03 16:01:35,627 [MainThread] INFO UNet3DTrainer - Logging model parameters and gradients
2023-02-03 16:01:36,745 [MainThread] INFO UNet3DTrainer - Training iteration [2501/60000]. Epoch [44/499]
2023-02-03 16:01:37,122 [MainThread] INFO UNet3DTrainer - Training iteration [2502/60000]. Epoch [44/499]
2023-02-03 16:01:37,647 [MainThread] INFO UNet3DTrainer - Training iteration [2503/60000]. Epoch [44/499]
2023-02-03 16:01:38,199 [MainThread] INFO UNet3DTrainer - Training iteration [2504/60000]. Epoch [44/499]
2023-02-03 16:01:38,762 [MainThread] INFO UNet3DTrainer - Training iteration [2505/60000]. Epoch [44/499]
2023-02-03 16:01:39,334 [MainThread] INFO UNet3DTrainer - Training iteration [2506/60000]. Epoch [44/499]
2023-02-03 16:01:39,882 [MainThread] INFO UNet3DTrainer - Training iteration [2507/60000]. Epoch [44/499]
2023-02-03 16:01:40,470 [MainThread] INFO UNet3DTrainer - Training iteration [2508/60000]. Epoch [44/499]
2023-02-03 16:01:41,010 [MainThread] INFO UNet3DTrainer - Training iteration [2509/60000]. Epoch [44/499]
2023-02-03 16:01:41,632 [MainThread] INFO UNet3DTrainer - Training iteration [2510/60000]. Epoch [44/499]
2023-02-03 16:01:42,223 [MainThread] INFO UNet3DTrainer - Training iteration [2511/60000]. Epoch [44/499]
2023-02-03 16:01:42,763 [MainThread] INFO UNet3DTrainer - Training iteration [2512/60000]. Epoch [44/499]
2023-02-03 16:01:43,325 [MainThread] INFO UNet3DTrainer - Training iteration [2513/60000]. Epoch [44/499]
2023-02-03 16:01:43,881 [MainThread] INFO UNet3DTrainer - Training iteration [2514/60000]. Epoch [44/499]
2023-02-03 16:01:44,422 [MainThread] INFO UNet3DTrainer - Training iteration [2515/60000]. Epoch [44/499]
2023-02-03 16:01:44,951 [MainThread] INFO UNet3DTrainer - Training iteration [2516/60000]. Epoch [44/499]
2023-02-03 16:02:19,371 [MainThread] INFO UNet3DTrainer - Training iteration [2517/60000]. Epoch [44/499]
2023-02-03 16:02:19,730 [MainThread] INFO UNet3DTrainer - Training iteration [2518/60000]. Epoch [44/499]
2023-02-03 16:02:20,266 [MainThread] INFO UNet3DTrainer - Training iteration [2519/60000]. Epoch [44/499]
2023-02-03 16:02:20,795 [MainThread] INFO UNet3DTrainer - Training iteration [2520/60000]. Epoch [44/499]
2023-02-03 16:02:28,542 [MainThread] INFO UNet3DTrainer - Training iteration [2521/60000]. Epoch [45/499]
2023-02-03 16:02:48,114 [MainThread] INFO UNet3DTrainer - Training iteration [2522/60000]. Epoch [45/499]
2023-02-03 16:02:48,487 [MainThread] INFO UNet3DTrainer - Training iteration [2523/60000]. Epoch [45/499]
2023-02-03 16:02:49,041 [MainThread] INFO UNet3DTrainer - Training iteration [2524/60000]. Epoch [45/499]
2023-02-03 16:02:49,622 [MainThread] INFO UNet3DTrainer - Training iteration [2525/60000]. Epoch [45/499]
2023-02-03 16:02:50,140 [MainThread] INFO UNet3DTrainer - Training iteration [2526/60000]. Epoch [45/499]
2023-02-03 16:02:50,791 [MainThread] INFO UNet3DTrainer - Training iteration [2527/60000]. Epoch [45/499]
2023-02-03 16:02:52,931 [MainThread] INFO UNet3DTrainer - Training iteration [2528/60000]. Epoch [45/499]
2023-02-03 16:02:53,370 [MainThread] INFO UNet3DTrainer - Training iteration [2529/60000]. Epoch [45/499]
2023-02-03 16:02:53,982 [MainThread] INFO UNet3DTrainer - Training iteration [2530/60000]. Epoch [45/499]
2023-02-03 16:02:54,531 [MainThread] INFO UNet3DTrainer - Training iteration [2531/60000]. Epoch [45/499]
2023-02-03 16:02:55,150 [MainThread] INFO UNet3DTrainer - Training iteration [2532/60000]. Epoch [45/499]
2023-02-03 16:02:57,721 [MainThread] INFO UNet3DTrainer - Training iteration [2533/60000]. Epoch [45/499]
2023-02-03 16:03:23,270 [MainThread] INFO UNet3DTrainer - Training iteration [2534/60000]. Epoch [45/499]
2023-02-03 16:03:23,778 [MainThread] INFO UNet3DTrainer - Training iteration [2535/60000]. Epoch [45/499]
2023-02-03 16:03:24,371 [MainThread] INFO UNet3DTrainer - Training iteration [2536/60000]. Epoch [45/499]
2023-02-03 16:03:24,950 [MainThread] INFO UNet3DTrainer - Training iteration [2537/60000]. Epoch [45/499]
2023-02-03 16:03:25,489 [MainThread] INFO UNet3DTrainer - Training iteration [2538/60000]. Epoch [45/499]
2023-02-03 16:03:26,107 [MainThread] INFO UNet3DTrainer - Training iteration [2539/60000]. Epoch [45/499]
2023-02-03 16:03:50,521 [MainThread] INFO UNet3DTrainer - Training iteration [2540/60000]. Epoch [45/499]
2023-02-03 16:03:50,919 [MainThread] INFO UNet3DTrainer - Training iteration [2541/60000]. Epoch [45/499]
2023-02-03 16:03:51,474 [MainThread] INFO UNet3DTrainer - Training iteration [2542/60000]. Epoch [45/499]
2023-02-03 16:03:52,028 [MainThread] INFO UNet3DTrainer - Training iteration [2543/60000]. Epoch [45/499]
2023-02-03 16:03:52,573 [MainThread] INFO UNet3DTrainer - Training iteration [2544/60000]. Epoch [45/499]
2023-02-03 16:03:53,156 [MainThread] INFO UNet3DTrainer - Training iteration [2545/60000]. Epoch [45/499]
2023-02-03 16:04:35,914 [MainThread] INFO UNet3DTrainer - Training iteration [2546/60000]. Epoch [45/499]
2023-02-03 16:04:36,300 [MainThread] INFO UNet3DTrainer - Training iteration [2547/60000]. Epoch [45/499]
2023-02-03 16:04:36,829 [MainThread] INFO UNet3DTrainer - Training iteration [2548/60000]. Epoch [45/499]
2023-02-03 16:04:37,381 [MainThread] INFO UNet3DTrainer - Training iteration [2549/60000]. Epoch [45/499]
2023-02-03 16:04:37,954 [MainThread] INFO UNet3DTrainer - Training iteration [2550/60000]. Epoch [45/499]
2023-02-03 16:04:38,498 [MainThread] INFO UNet3DTrainer - Training iteration [2551/60000]. Epoch [45/499]
2023-02-03 16:04:40,521 [MainThread] INFO UNet3DTrainer - Training iteration [2552/60000]. Epoch [45/499]
2023-02-03 16:04:40,926 [MainThread] INFO UNet3DTrainer - Training iteration [2553/60000]. Epoch [45/499]
2023-02-03 16:04:41,525 [MainThread] INFO UNet3DTrainer - Training iteration [2554/60000]. Epoch [45/499]
2023-02-03 16:04:42,118 [MainThread] INFO UNet3DTrainer - Training iteration [2555/60000]. Epoch [45/499]
2023-02-03 16:05:02,523 [MainThread] INFO UNet3DTrainer - Training iteration [2556/60000]. Epoch [45/499]
2023-02-03 16:05:02,928 [MainThread] INFO UNet3DTrainer - Training iteration [2557/60000]. Epoch [45/499]
2023-02-03 16:05:03,492 [MainThread] INFO UNet3DTrainer - Training iteration [2558/60000]. Epoch [45/499]
2023-02-03 16:05:06,554 [MainThread] INFO UNet3DTrainer - Training iteration [2559/60000]. Epoch [45/499]
2023-02-03 16:05:07,035 [MainThread] INFO UNet3DTrainer - Training iteration [2560/60000]. Epoch [45/499]
2023-02-03 16:05:07,589 [MainThread] INFO UNet3DTrainer - Training iteration [2561/60000]. Epoch [45/499]
2023-02-03 16:05:31,030 [MainThread] INFO UNet3DTrainer - Training iteration [2562/60000]. Epoch [45/499]
2023-02-03 16:05:31,798 [MainThread] INFO UNet3DTrainer - Training iteration [2563/60000]. Epoch [45/499]
2023-02-03 16:05:32,168 [MainThread] INFO UNet3DTrainer - Training iteration [2564/60000]. Epoch [45/499]
2023-02-03 16:05:32,700 [MainThread] INFO UNet3DTrainer - Training iteration [2565/60000]. Epoch [45/499]
2023-02-03 16:05:33,264 [MainThread] INFO UNet3DTrainer - Training iteration [2566/60000]. Epoch [45/499]
2023-02-03 16:05:36,394 [MainThread] INFO UNet3DTrainer - Training iteration [2567/60000]. Epoch [45/499]